Tukukkk

Let's say I have a list of images (converted to numpy arrays) downloaded from Instagram, along with their corresponding likes and user followers. And let's say I have a CNN model (using Keras on Tensorflow) which I train on these images (200x200x3 numpy arrays) and it tries to predict the number of likes an image will get.

What if I want to give to this model each image's corresponding followers as a second input?

This is my code so far:

IMAGESIZE = (200, 200)



def create_model():

    # create model and add layers

    model = Sequential()



    model.add(Conv2D(10, 5, 5, activation='relu',

                     input_shape=(IMAGESIZE[0], IMAGESIZE[1], 3)))



    model.add(Conv2D(10, 5, 5, activation='relu'))

    model.add(MaxPool2D((5, 5)))

    model.add(Dropout(0.2))

    model.add(Flatten())

    model.add(Dense(50))

    model.add(Activation('relu'))

    model.add(Dense(1))



    print(model.summary())



    model.compile(loss='mse',

                  optimizer='rmsprop', metrics=["accuracy"])

    return model



# Read the likes

likes = getlikes(src='../data/pickledump')

likesraw = np.array(likes)

likes = (likesraw - np.mean(likesraw))/np.std(likesraw)  # normalize



# Read the images and resize them

images = 

for imgfile in glob.glob('../data/download/*.jpeg'):

    img = cv2.imread(imgfile)

    resized = cv2.resize(img, IMAGESIZE)

    images.append(resized)

    break

images = np.array(images)



# Read the followers

followers= getfollowers(src='../data/pickledump')

followersraw= np.array(followers)

followers= (followersraw- np.mean(followersraw))/np.std(followersraw)  # normalize



classifier = KerasClassifier(build_fn=create_model, epochs=20)

print("Accuracy (Cross Validation=10): ",

      np.mean(cross_val_score(classifier, images, likes, cv=2)))

One approach is to use a two branch model, where one branch processes the image and another branch processes other non-image inputs (such as posts texts or number of followers and followings, etc.). Then you can merge the result of these two branches and possibly add a few other layers afterwards to act a the final classifier/regressor. To build such a model in Keras you need to use the functional API instead. Just for demonstration, here is an example:

inp_img = Input(shape=image_shape)

inp_others = Input(shape=others_shape)



# branch 1: process input image

x = Conv2D(...)(inp_img)

x = Conv2D(...)(x)

x = MaxPool2D(...)(x)

out_b1 = Flatten()(x)



# branch 2: process other input

out_b2 = Dense(...)(inp_other)





# merge the results by concatenation

merged = concatenate([out_b1, out_b2])



# pass merged tensor to some other layers

x = Dense(...)(merged)

output = Dense(...)(x)



# build the model and compile it

model = Model([inp_img, inp_other], output)

model.compile(...)



# fit on training data

model.fit([img_array, other_array], label_array, ...)

Note that we used concatenation layer above, but there are other merge layers which you can use. And make sure you read the functional API guide, it's a must-read guide.