pytoch如何加载批量数据--Dataset，DataLoader

本文主要是介绍pytoch如何加载批量数据--Dataset，DataLoader，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

Dataset

Dataset 是一个抽象类。我们可以定义一个类继承这个类，从而加载数据，构造数据集（索引）

DataLoader

DataLoader是一个帮助我们在Pytorch中加载数据的类，在训练测试时加载数据，获取mini-batch

使用说明：¶

epoch：One forward pass and one backward pass of all the training examples. 训练次数

batch-size：The number of tarining examples in one forward backward pass. 每次用的样本数量

iterations：Number of passes，each pass using [batch size] number of examples. 迭代的数量 样本数量/batch

import torch
from torch.utils.data import Dataset
from torch.utils.data import DataLoader
import numpy as np
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

准备数据： 实现三个魔法方法

# 继承抽象类Dataset，实现三个方法
class DiabetesDataset(Dataset):# def __init__(self,filepath):#     data =  np.loadtxt(filepath,delimiter=',',dtype=np.float32)#     self.len = data.shape[0]  # (d0,d1,d2..) d0#     self.X = torch.from_numpy(data[:,:-1])#     self.Y = torch.from_numpy(data[:,[-1]])# 1.所有数据一次性加载# 2.每次只加载batch的数据def __init__(self,data,label):self.len = data.shape[0]self.X = torch.from_numpy(data)self.Y = torch.from_numpy(label)# 实例化类后，使对象支持索引操作 The expression，dataset[index],will call this magic functiondef __getitem__(self,index):return self.X[index], self.Y[index] # 返回元组 # this magic function returns length of datasetdef __len__(self):return self.len# 划分训练集和测试集    
data =  np.loadtxt('diabetes.csv',delimiter=',',dtype=np.float32)
X = data[:,:-1]
Y = data[:,[-1]]
# train_data,test_data = train_test_split(data,test_size=0.2,random_state=42)
Xtrain,Xtest,Ytrain,Ytest = train_test_split(X,Y,test_size=0.2)# 实例化dataset类
dataset = DiabetesDataset(Xtrain,Ytrain)
# DataLoader：加载器 Initialize loader with batch-size，shuffle，process number
train_loader = DataLoader(dataset=dataset,batch_size=15,shuffle=True# ,num_workers=2)
# num_workers 线程，并发数，用了发现超conda虚拟内存

diabetes.csv糖尿病数据集

        链接：https://pan.baidu.com/s/1a-6ToVlXr7QfYAnHIpWnNg?pwd=1234 
        提取码：1234 

设计模型：

class Model(torch.nn.Module):def __init__(self):super(Model,self).__init__()self.linear1 = torch.nn.Linear(8,6)self.linear2 = torch.nn.Linear(6,4)self.linear3 = torch.nn.Linear(4,1)self.sigmoid = torch.nn.Sigmoid()def forward(self,x):x = self.sigmoid(self.linear1(x))x = self.sigmoid(self.linear2(x))x = self.sigmoid(self.linear3(x))return xmodel = Model()

构造损失函数和优化器

criterion = torch.nn.BCELoss(size_average=True,reduction='mean')
optimizer = torch.optim.SGD(model.parameters(),lr=0.01)

训练

# if name==__main__():
for epoch in range(10000):for i,data in enumerate(train_loader,0):# for i,(x,y) in enumerate(train_loader,0):inputs,labels = datay_pred = model(inputs)loss = criterion(y_pred,labels)# print(epoch,i,loss.item())optimizer.zero_grad()loss.backward()optimizer.step()

测试

x_test = torch.from_numpy(Xtest)
y_test = torch.from_numpy(Ytest)
y_pred = model(x_test)
y_pred_label = torch.where(y_pred>=0.5,torch.tensor([1.0]),torch.tensor([0.0]))
acc = torch.eq(y_pred_label, y_test).sum().item()/y_test.size(0)
print("acc = ",acc)

这篇关于pytoch如何加载批量数据--Dataset，DataLoader的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！

---