\n \nHmm, how did those golden ratio rules go again?
\n\nThis is be a nice little post about a neural network.
\nI like to periodically spend my free time behind the camera. Capturing moments, entities, phenomena and ideas is extremely rewarding, especially when you get that perfect shot. What isn't rewarding, however, is the time spent on post-processing the images; adjusting the abundant options, tweaking the plethora of configurations and trying to decide the perfect ensemble for a shot.
\nThis, plus a combination of my innate laziness and curiosity, inspired me to create a small neural network to assist with the process.
\n
\n \n
\nI realised that I could extract information on the photo edits from the metadata added to the photo files using Exchangeable image file format (Exif), and use that as training data. To make matters a bit easier and the scope of the project a bit smaller, I decided to limit the edits to specifically cropping (and small rotational alignment).
\n\n \n \n
\nHmm, how did those golden ratio rules go again?
Firstly, I wrote a small python command line tool to easily crawl my photo library for possible images, i.e. all the photos that had been cropped. I explained this in more details in the ExifAnnotator post.
\nEssentially, I could give the tool a path as an argument, and it would quickly find the photos which had been edited, resized them to a predefined resolution (while keeping the network training in mind) and put them into a predefined folder, as well as collected a nice json label-set for said photos. All good to go! Making my dataset was practically done! Moreover, should I need more photos, I could just run the script again for a separate folder.
\nFurthermore, I could expand the edits by simply referring to the Exif notation and defining the values I wanted to extract. Sweet.
\nUsing the extracted data, we can see that a crop on an image is defined as four values, defining the limits from top, left, bottom and right. These limits are defined as fraction fo the image, i.e. 0-1, 0 being the leftmost and topmost edges, and 1 correspondingly the rightmost and bottommost edges. Additionally I included a value describing a rotation on the image, positive value describing a rotation clockwise, and negative anti-clockwise.
\n| Metadata tag | \nExample value | \n
|---|---|
XMP:CropTop | \n0.217628 | \n
XMP:CropLeft | \n0 | \n
XMP:CropBottom | \n0.9646 | \n
XMP:CropRight | \n1 | \n
XMP:CropAngle | \n-0.52 | \n
An example of the crop defined by the metadata tags. The photo in question has been cropped in the vertical direction, and tilted slightly anti-clock-wise.
\nAlready we have the values which we want to predict, a vector containing five values. Excellent! Moreover, all the values are neatly within the range 0-1, and can be easily normalised to -0.5 - 0.5. to achieve zero-mean.
\n\n \n
\nHow do you crop your crop?
Now, I really enjoy tackling problems head on. Thanks to that, I have ended up re-inventing the wheel several times over. Naturally, this is something I have learned to avoid by putting more effort in background research and utilising existing solutions.
\nNevertheless, I still find trying to solve a problem on my own extremely interesting - without any help from \"how it is done the best way\" -guides. Simply breaking the problem down and working on your own solution can help you gain knowledge and deeper understanding on the topic; knowledge that you would miss should you simply follow a \"proven\" solution.
\nNaturally, such luxuries are rare in a world that demands fast pace and high quality. Since this was not the case on this project, I jumped straight in!
\nNaturally, as we are processing images, I chose to use a CNN. I started with a simple one containing only a few layers, eventually leading to a fully connected layer which would output the prediction.
\n\n \n 
\n
\nThe very basic structure of any CNN: convolutions and subsamplings
Along with the basic CNN building blocks, subsampling and convolutional layers, I eventually added batch normalisation and dropout layers to mitigate over-fitting. After all, I was using a relatively small set of training data. The final architecture of the network was obtained with trial and error: using Tensorboard as a tool to observe the performance of the network and the effect of the changes made.
\nI went with the simplest initial plan there was: using a Square Mean Error (SME) for the predictions. I wanted to see how well I could make the network run without introducing anything fancy, such as area overlap used in many object detection methods.
\n![\"The](\"/static/tb_ext-e41a2323e6e474ac3464bcb862faa2dc.gif\")
\nCrops for unseen data during training for each completed epoch, using a custom Tensorboard callback
I had noticed the new release of TensorFlow 2.0 and was keen on trying it out. I set up some draft jupyter notebooks with TensorFlow with CUDA acceleration installed, and started developing. I also whipped up a small script to visualise the progress on evaluation data in TensorBoard.
\nThen, I just started iterating.
\n\n\nNow, first I need to address the issue of the network design. Considering the challenge and my lack of experience on CNN for such applications, I didn't go for any specific architecture. I started reading basic articles on image processing and modified the network accordingly. In other words, this could had been much more sophisticated and probably will make a couple sworn deep learning peeps pull a healthy handful of hair out of their head.
\n
Next, I'll describe a few problems I ran into and how I tried to fix them.
\nSo, it turns out that the network thinks that there is interesting stuff happening outside the photo boundaries as well.
\n\n \n 
\n
\nThe CNN low-key trying to hint that you should invest in a wide-angle lens
As technically the values for the prediction were not limited in any manner, they soon went ballistic. Initially I had hoped that the training data would be enough to limit the values but obviously not. Hence I figured there are three things that I could do:
\nAs getting more samples was a limited option, I tried to create a custom loss function which would, by some scaling, add error if the values were outside the set limits.
\n # Clip constraint:\n clip_constraint = BK.sum(BK.square(y_pred * BK.cast(tf.math.logical_or(\n BK.less_equal(y_pred, 0.0), BK.greater_equal(y_pred, 1.0)\n ), BK.floatx())), axis=-1)*Snippet of the custom loss. BK is the Keras backend. Full module can be found in the GitHub repository*
This, however did not yield any better results. Moreover, I could simply clip the resulting values to a certain limits after obtaining the result, so I did not spend too much time thinking of improvements. Maybe it would be beneficial to circle back to this problem in the future, as I find many of real world problems are innately somehow limited.
\nAnother problem arising from how I used the predictions was that the values could overlap, making the area non-existent. This, however, was a relatively rare case and seemed to go away with more training iterations.
\nI wonder, though, would the use of another type of error criterion help with this?
\nOn practically every initial run, the graphs showing the accuracy and loss on familiar and unseen data fed to the network were not that satisfactory of a shape: the network was clearly overfitting the data... No wonder the photos looked so nice in the end.
\n\n \n 
\n
\nCNN: \"If something is worth fitting, it is worth overfitting\" Orange being the training data, blue the evaluation data.
To combat this, I added some dropout and batch normalisation layers to my network, as well as made it deeper and more narrow. However, nothing beats having more data.
\n\n \n 
\n
\nThe crop values used in training heavily slanted statistically, causing possible probability bias to predictions.
Additionally, the data was quite heavily slanted towards crops on the top left corner, and rather large areas instead of a clear, small areas. This was naturally the result of my cropping style, so it was also interesting to learn a bit about myself. However, some data augmentation could had been added to normalise the distributions of the training data fed to the network, such as horizontal flipping.
\nAfter numerous iterations on the network architecture and loss functions, I ended up with the following architecture:
\nModel: \"sequential_1\"\n_________________________________________________________________\nLayer (type) Output Shape Param # \n=================================================================\nconv2d_6 (Conv2D) (None, 298, 298, 64) 1792 \n_________________________________________________________________\nmax_pooling2d_4 (MaxPooling2 (None, 149, 149, 64) 0 \n_________________________________________________________________\nconv2d_7 (Conv2D) (None, 147, 147, 64) 36928 \n_________________________________________________________________\nmax_pooling2d_5 (MaxPooling2 (None, 73, 73, 64) 0 \n_________________________________________________________________\ndropout_2 (Dropout) (None, 73, 73, 64) 0 \n_________________________________________________________________\nconv2d_8 (Conv2D) (None, 71, 71, 64) 36928 \n_________________________________________________________________\nmax_pooling2d_6 (MaxPooling2 (None, 35, 35, 64) 0 \n_________________________________________________________________\nconv2d_9 (Conv2D) (None, 33, 33, 64) 36928 \n_________________________________________________________________\nmax_pooling2d_7 (MaxPooling2 (None, 16, 16, 64) 0 \n_________________________________________________________________\ndropout_3 (Dropout) (None, 16, 16, 64) 0 \n_________________________________________________________________\nconv2d_10 (Conv2D) (None, 14, 14, 64) 36928 \n_________________________________________________________________\nconv2d_11 (Conv2D) (None, 14, 14, 64) 4160 \n_________________________________________________________________\nflatten_1 (Flatten) (None, 12544) 0 \n_________________________________________________________________\ndense_3 (Dense) (None, 64) 802880 \n_________________________________________________________________\ndense_4 (Dense) (None, 32) 2080 \n_________________________________________________________________\ndense_5 (Dense) (None, 5) 165 \n=================================================================\nTotal params: 958,789\nTrainable params: 958,789\nNon-trainable params: 0\n_________________________________________________________________The final architecture of the network
\nFor loss, I found the basic SME to work best. Generally, the resulting crops seemed sensible, and more importantly, deterministic based on the input image. I was satisfied, considering my skills and the time I had allocated for this project in between my studying and work.
\nHowever, as one can obviously tell, the resulting crops are rather, well, open for interpretation: had the network really learnt to crop the images, or simply pseudo-randomly sprouted predictions, using the input image as a seed and the training crop distributions as the value pool? Especially, the values seemingly restricted to [|0.6, 1|], practically any random crop could be interpreted by a user as a \"sensible crop\", especially if the image was already at least somehow sensibly cropped to begin with.
\nIt would be indeed interesting to have a comparison of the different methods: one image, cropped by a human, a trained CNN and a random generator with sensible limits. Would a person be able to tell a difference? What would it prefer? Could they be improved? Could any of these systems learn from each other? Is there something to be learnt from the expression \"beauty is in th eye of the beholder?\"
\n\n \n 
\n \nSome example predictions.
Naturally I wanted to show off with this cool little toy I had created. I did not want to drag my laptop around or record hours on my screen - nah, I'd rather whip up a simple tool to let people play around with the network. Sharing is caring after all, right?
\nI was also interested in the at the time recent TensorFlow.js (TensorFlow JavaScript) library. Apparently with this, it would be rather straightforward to add your networks to web applications.
\n \n 
\n \nThe UI for the CropperHead \"Service\"
By following the few tutorials available on tfjs website, I managed to pull together a rather decent looking website. Simple enough, no extra shinies and included all the necessary funcitonality: the user could select a photo from their computer to be analysed and see the estimated values. I used `cropper.js for displaying the estimated crop on the web application. However, as the cropper.js had such nice functionality for cropping the photo, I was inspired...
As quite obvious from the text above, the implementation of the network was not by any means optimal. On top of the shortcomings I already listed, one major flaw also existed: lack of training data. Despite my efforts of crawling through the photo library I accumulated over the years, as well as trying to crowd-source data collection to my friends who also enjoy photography, I only managed to collate around 300 photos. Considering the problem, I would estimate to need around ten times more... [LINKS TO ARGUMENT] So, how to get it?
\nThe obvious solution to collect the data in form and without constraints, would be to utilise the UI! As people would be able to add their own photos, and cropper.js supported cropping on the UI, it would be quite convenient to combine these two for a pipeline to collect the much needed data. A simple block of text assuring our benevolent means and needs, along with a hint of instructions, and a coule of buttons later, the UI was ready to process the preferred crops of the users!
\n \n 
\n \nHow to tell me how to do my job
The UI side was fixed, bu what about the server side? How, where and when were the photos and labels processed? As I had some previous experience using mongoDB, AWS and microservices, I implemented a (not so simple) scalable data collection service - DataLibrarian! With it, I could easily set up a service which would nicely route the photos to a AWS S3 bucket, the lables to mongoDB and keep a record of how they were paired. YOu can read more about the project in its dedicated project description.
\n \n 
\n \nSimplified view of the entire project architecture
So, with the few extra hours I had for myself after work, I had managed to effectively collect training data by extracting crop information from Exif, create and train a Convolutional Neural Network to crop (and slightly rotate) photos automatically, design and publish a web application where users all around the world could try out the performance of the network, along with a method and infrastructure for collecting more training data to further improve my model!
\nI am rather happy with the overall result. All of the individual block can naturally be improved (quite a lot) but also the entire approach could be different. For example, the following might be interesting changes:
\nNevertheless, the entire project served its purpose as an entertaining way to try and learn new. Implementing such a solution from start to finish hopefully helps me to extensively learn and communicate with appropriate professionals, and concentrate on more pressing issues. Such wonderful time!
"}},"pageContext":{"slug":"cropperhead","navContext":{"next":{"path":"/projects/exifextractor","title":"ExifExtractor - Find your edits","slug":"exifextractor","links":["https://www.github.com/PebbleBonk/ExifAnnotator"]},"prev":{"path":"/projects/texmindmapper","title":"TexMindMapper - visualise your thesis","slug":"texmindmapper","links":["https://www.github.com/PebbleBonk/TexMindMapper","https://tex-mind-mapper.herokuapp.com"]}},"links":["https://www.github.com/PebbleBonk/CropperHead","https://www.github.com/PebbleBonk/CropperHeadUI","https://cropper-head.herokuapp.com"]}}}