Going serverless: Part 1 - Introduction

Introduction

I've been on a pretty heavy Serverless kick over the last 12 months. At my day job and at home, I've started all of my projects with Serverless being the default deployment plan. It's simplified things so much that I've abanonded .NET development for back-end services in favor of Node.js because the Serverless containers start faster with it.

Making the shift to Serverless compute came with a lot of new terms and technologies that I had to learn. Moving to NoSQL Document databases, Node, CLI driven work-flows and the most difficult being developing products outside of the glorious debugging experience I've had with Visual Studio.

To that end, I've decided to blog a complete guide to going Serverless. The guide will be split over multiple posts, from creating a cloud account, to setting up a local development environment, debugging and deploying the service. If time allows I might explore deploying a web-site as a serverless solution as well. For the initial posts in this series though I will be focusing on the creation and deployment of back-end API services.

Each post in the series can be accessed here:

What is Serverless?

Over the last few years a new technology has slowly gained popularity in the operations support world. The new technology provided a service for operation admins, security engineers, DevOps engineers and developers to build jobs that ran in the cloud without needing any infrastrucure. The service was called Functions as Service (FaaS) and allowed for deploying, literaly functions, into the cloud that could be executed.

Once deployed function could be triggerd by an event in the cloud. Such as uploading a file, deleting a file, a new message landing in a message bus, a configuration rule being violated etc. Even better was that FaaS allowed for the deployment of these functions without the engineer needing to provision a server for the function to run on. Depending on what your FaaS provider of choice is, the security model for the deployed function is incredibly simple as well. In some cases they can run outside of your network, allowing for them to run in isolation to your production environment. This reduces the blast-radius in the even that your function could potentially be compromised.

With FaaS working so well it was only a matter of time before functions integrated themselves with the web. Serverless compute was born when FaaS could be integrated behind API Gateways, allowing the functions to receive HTTP requests. Now FaaS can be triggered by HTTP requests, handle the request and send a response back to the original requestor. This marrying of technologies is where the term Serverless came from as there are no web servers for the developers to deploy to, no Apache/NGINX/IIS server for IT Administrators to patch and secure. Developers can write the web application with just a few small changes and deploy in seconds without any supporting infrastructure.

The cloud

For this series of post we will be focusing completely on Amazon Web Services (AWS) as our cloud back-end. Amazon's FaaS solution is called Lambda. Coupling Lambda with their API Gateway service provides you with a powerful and simple Serverless solution. Everything we will create in this example falls under Amazon's free tier, meaning for the first year you won't have to pay for anything. You are given 1,000,000 executions of your Lambdas for free, per month. You are given 25 reads and 25 writes per second on their NoSQL Document database called DynamoDB. That's way more than what we need to build out a development environment and deploy it.

We will provision our entire back-end using code as well. Creating your cloud environment manually through the AWS web site is prone to misconfiguration as you spin up new environments. Dev and QA will eventually drift from each other, and production and Dev will almost always start to look different as soon as you've deployed. Creating your resources via code allows for versioning your infrastructure. Having your infrastructure versioned makes roll backs a lot easier and enables the option for you to rapidly spin up additional resources on-demand if needed.

Wrapping up

Over the next few blog posts we will build our a simple Todo API that will not need a SQL store, won't need a web server and will provide free user provisioning and account authorization.

Let's get this party started, on to Part 2!

Wireless, cross-device, photo syncing in 2017

in

I've been meaning for a long time now, to share what I'm doing in regards to synchronizing photos across all of my devices. I originally posted back in 2014 how I was doing it, and again in 2015 with updates as technology changed. In 2016, my photo synchronization strategy changed once again, but I never got around to writing a post on it. So here I am, in 2017, sharing what's changed. As is usually the case with these posts - it's hefty. So get yourself your favorite beverage and snack!

What's changed?

Before we get into the nitty-gritty details, lets take a look at what i've changed first.

From a sync perspective, I've stopped using iCloud. It was a huge pain to deal with when trying to manage my photos in Lightroom. Exporting them from iCloud Photo Library, into Lightroom, tweaking them etc, and then re-importing the photos back into iCloud was extremely tedious and time consuming. Because of this, I found that I hardly ever actually migrated photos and it made it really hard to go back and find photos over time. Remember, this was early to mid 2016 - so iOS and macOS Photo apps were not as smart as they currently are now.

Ultimately, I setup a Google Photos library on my Google account. I then synced my entire Lightroom library into Google Photos. This worked amazingly well. The iOS Google Photos app would automatically upload my new photos in the background, and they'd sync down to my computers into my Lightroom library folder - ready for me to just perform "add missing files" in Lightroom. Any changes I made to the photos were immediately synced to all devices, because the library and photo lived in the synchronized Google Photos folder. The setup for this is explained in detail further below.

The biggest thing however is that I've stopped using Lightroom for photo organization. That does not however mean that my current solution wouldn't work with Lightroom. I was using Lightroom when I changed my syncing solution, and it worked without a problem. I just found that Google did such an amazing job with searching images, that I no longer needed to spend so much time organizing them myself. I now let Google Photos handle all of my organization needs. I explain this in more detail below.

The biggest gain from this change, for me, is that now my photos are synced across all devices, even those that are not Apple. We have an Android device in the house, and a couple Windows machines. All of which have my photos synced to. My wife has the Google Photos app installed on her iPhone and iPad, connected to my account. So her photos now sync into my Google Photos library (and at one point my Lightroom library). With virtually no management on my end, I'm syncing photos from 2 completely different Apple accounts, into a single consolidated family photo library. That's a big win for me.

Moving to Google Photos

When moving to Google Photos, there are a couple different routes you can take. It depends on what your end goals are.

The free solution

If you are not currently using any app, like Aperture or Lightroom, to manage your photos then setting up Google Photos is easy. You just install the Google Photos iOS or Android app, and configure it to automatically back up your photos. On iOS, it looks like this:

When you turn on Back up & sync, Google Photos on iOS will automatically back up your photos for you to the Google Photos service. Google Photos gives you unlimited photo uploads for pictures that are 16 megapixels or less. All of the current iPhones on the market take photos at less than 16 megapixels, so you can use the service for free with no limits. Google Photos will also let you store unlimited 1080p videos as well. Since some iPhones on the market can record 4K video, you would want to choose whether or not you want to record in 1080p so you can sync them for free, or record them in 4K and upload until you run out of storage space, which is 15gb on the free account.

If you go this route, all you have to do is install and sign in to your Google Photos account on all of your phones and tablets. Then you can access the photos from the Google Photos website on your computer.

The integrated Desktop solution

If you are using an app to organize your photos, or are wanting to start using one, then you'll want to get this photos synchronized to your desktop computer. This is extremely easy to do, at the cost of paying a monthly subscription to Google Drive. I currently pay $10 per month, for 1 terabyte of storage. Once you've done that, you can go to your Google Photos settings, from the website, and turn on Google Drive.

Once you have turned on Google Drive support in your Photos settings, you can install the Google Drive app on your macOS or Windows computer. Once it is installed, it will automatically sync all of your photos from Google Photos to your computer. Here you can see that my original Lightroom based organization is still intact, even though these photos exist now in Google Photos.

Since I stopped managing photos myself, any new photo I add to Google Photos gets put under the Google Photos folder, within a folder indicating the year the photo was created. This is why you see Google Photos\2010, and Google Photos\Digital Photos\10s\2010. Google will at least organize your photos by year for you automatically, if you choose not to use any software to manually handle this.

With this, you now have every photo you take, regardless if it's an iOS or Android device, synchronized to your desktop computer. From there, you can open Lightroom, select the Google Photos directory and have it import all new Photos for you. Once they're imported, any changes made to the photos, (including their file location as long as it's someplace under the Google Photos folder) will be synced to all of your devices automatically.

Leaving Lightroom

So now that we've talked about my synchronization solution, let me spend a couple minutes explaining why I moved away from Lightroom all together for my organizational needs.

The biggest reason is time. I don't have much of it, and I have to set aside a block of time to organize several hundred, or a thousand, photos every couple of weeks. My organization routine typically included moving the photos into sub-folders representing the activity that took place, tagging every photo and confirming facial recognition. This usually took a minimum of two hours for me.

Enter Google Photos. They do most of this for me. The only exception is that they organize the photos under a single folder, representing the year. Where-as I would break the photos up under a year, by month. Here you can see Google Photos organization of the photos since I stopped doing it myself, for 2017.

They organize them in a single directory for the entire year. When I used to organize them in Lightroom, I organized them by year, month and activity as seen here.

I now just use Windows Explorer to move specific sets of Photos into sub-folders if I need to. Any photo that is stored within a sub-folder of the year, will let Google Photos create an album for it. For instance, in my example photo above, I have a folder called eating at Islands Burgers. Google turned that into an album for me, so when I search for Islands Burgers in Google Photos (regardless if it's the mobile app or website) their auto-complete will know what you mean and present it as an option.

This to me is pretty awesome. My custom organization strategies can be persisted across all of my devices using Google Photos this way. So why pay for Lightroom for organizing? Just organize them in Windows Explorer if you want. If I were to select the Eating at Islands Burgers album suggested to me, I would be presented with the album (represented as a folder on my computer) as shown below.

Now, Lightroom does some other things - such as tagging your photos. Unfortunately Google Photos doesn't read your tags or support custom tagging of photos. Here's the crazy thing though. They scan the photos for content and automatically associate tags for you. From my experience with it, it does a far better job of it than I ever did. For example, I can search for all photos of Batman - and get back comic books and clothes. None of these photos were tagged by me in Lightroom. Even the blanket in the photo has batman in it!

Another example is searching for "Zoo", which presented me with photos both my wife and I had taken at the various zoos we've been two. It also included two different albums we had created with photos from different occasions.

I've searched for Nintendo controllers, Monitors, 2006 Scion Xb, Chihuahua and have been given every photo containing these items. It is amazing when you experience it for the first time. Once I used it enough to see how accurate and thorough Google Photos tags stuff, I stopped worrying about tagging things myself.

Lightroom supports facial recognition, and so does Google Photos. Google Photos does an amazing job of it though. I synced photos of our newborn daughter and assigned her a face in Google Photos. From that point forward, as we take pictures of her it automatically assigns her face to the photos. Even though she's now 3 years old and looks completely different from the 1 hour old baby I first uploaded a picture of, Google Photos has kept up and figured it out. Since Google Photos handles this, I stopped using this feature in Lightroom.

The last thing that Lightroom is good at is editing the actual photos. Google Photos has some basic tools, like crop, rotate etc. It even has a few filters you can apply. It's not the powerhouse that Lightroom is though, so if you find yourself editing photos often, you'll still want to keep doing that in Lightroom. This is something I hardly ever did, so it was just one less reason for me to keep paying for Lightroom.

Even though Google Photos can automatically do a lot of what you have to do manually in Lightroom, there isn't anything stopping you from continuing to use Lightroom.

Google Photos Assistant

Google Photos has a feature called the Assistant which will automatically take photos you've uploaded and turn them into albums, collages, home movies and animated gifs. It's really awesome. For example, it took a series of photos over the course of two weeks of my son and organized them into an album and collage for me called Highlights of Oliver

Selecting the Highlights of Oliver tile opened up the collage so that I can vew around 20 photos of him over a span of two weeks. Photos taken by both myself and my wife, which is really a nice touch.

Google Assistant on Android

The last thing that's neat, if you are on an Android device that supports the Google Assistant, is that the Google Assistant will search your photos too. Here I asked the Assistant to find photos from a park, and it knew not to just do a Google search of park pictures. Instead, it searched my personal photo collection and find some photos. If I touched the Google Photos button, it would open the app and give me a wider selection of images to look through.

Wrapping up

I've used this system for over a year now and love it. It has simplified the way I do photo synchronization and management across all of my devices, and my wifes devices.

Lightmap for .Net

in

I've been working on a new open source project called Lightmap that provides developers with a way to model their databases, in a database agnostic manor. This is done by creating migration objects that define what the table layout (along with other database objects) would look like. Lightmap supports several different ways of doing this - the following example shows just one. We'll use a couple different C# model classes to represent tables, related to each other.

Migrations

We'll use the following two classes as our table definitions.

public class User
{
    public int UserId { get; set; }
    public string Email { get; set; }
    public int LockoutEnabled { get; set; }
    public string PasswordHash { get; set; }
    public string UserName { get; set; }
}

public class UserLogins
{
    public string LoginProvider { get; set; }
    public int ProviderId { get; set; }
    public string ProviderDisplayName { get; set; }
    public string UserId { get; set; }
}

Next we'll create our database model with an initial migration object.

public class InitialMigration : IMigration
{
    public InitialMigration(IDataModel dataModel) => this.DataModel = dataModel;

    public IDataModel DataModel { get; }

    public void Apply()
    {
        this.DataModel.AddTable<User>()
            .AlterColumn(table => table.UserId).IsPrimaryKey();

        this.DataModel.AddTable<UserLogin>()
            .AlterColumn(table => table.ProviderId).IsPrimaryKey()
            .WithForeignKey<User>(
                (userTable, loginTable) => userTable.UserId == loginTable.UserId);
    }

    public void Revert() { } // Revert support not implemented yet.
}

The API is fairly straight forward, and has hook points for database provider specific implementations to expose additional constraints and database features via extension methods.

Wiring it up

To get it wired up in your application, you need to create an instance of the IDataModel implementation, IDatabaseManager implementation for the database of your choice, along with an instance of an IDatabaseMigrator that works with your database.

Currently only Sqlite has working implementations. The Lightmap API and base feature set is still being developed; until it's stabilized, I won't be adding other databases myself. Sql Server and MySql are the next two I plan on supporting as soon as the API churn quiets down.

The wire up for Lightmap will be simplified in the future as well, for apps that use ASP.Net Core. For now, you would wire it all up manually like this:

IDataModel dataModel = new DataModel();
IDatabaseManager databaseManager = new SqliteDatabaseManager("MyDb", "Data Source=MyDb.sqlite");
IMigration migration = new InitialMigration(dataModel);
IDatabaseMigrator migrator = new SqliteMigrator(migration);
migrator.Apply(databaseManager);

Ideally, all you would have to do is use the ASP.Net Core middleware to wire this up. I'm thinking of something along the lines of:

appBuilder.UseLightmap(config => config.WithSqlite());

There will be additional things on the configuration callback instance that you can use to setup various options, like default schema names.

Wrapping up

The project is still under active development and isn't ready for a release yet. If anyone would like to take it for a spin, they can clone it from GitHub, compile and add it to their projects. It currently only runs on NetStandard 1.4. The target framework will probably change to NetStandard 2.0 when that ships and I'll also work towards supporting Net46 as well so that the library can run on Full Framework, Mono and Core.

I'm not to far out from having an initial beta shipped on NuGet.

Wireless, Cross-device, photo syncing in 2015

in

Once again I'm visiting my work flow for syncing photos wirelessly from my phone, to my computer and back to the phone. I have blogged about this a few times in the past. In my last blog post regarding photo sync, I had turned to Lightroom mobile. It worked well for photos, but lacked any kind of support for videos.

A lot has changed over the last year. Now both Apple and Google provide bi-directional, wireless, sync of both photos and videos to and from your mobile devices, while keeping originals synced on your desktop computer and stored in the cloud. Take a photo or record a video on your device, open your computer to edit it, view the changes later in the day on your phone. It's pretty slick.

Now that both companies support sync in this manor, I thought I'd go over my workflow. The changes in the way both services handle the sync have caused me to change my work flow a bit. Since I use iOS and OS X devices most, I will be writing about Apples services today. At some point in the future I'll try and write something up on using Google Photos + Google Drive on Macs and iOS for cross-device syncing of photos and videos.

iCloud Photo Library

In 2014 Apple introduced the new iCloud Photo Library and Photos app in iOS 8. The iCloud Photo Library handles synchronizing all of your full resolution photos and videos across every iOS and OS X based device you own. Unfortunately, this is not available on AppleTV.

So, we need to turn on iCloud Photo Library on the iOS devices. You can do that by going into the iOS Settings app and navigating to Photos & Camera.

I typically leave my iOS devices with Optimize iPhone Storage enabled. This will upload the original files to the iCloud Photo Library; saving a reduce size version for the smaller iOS screens on the device. In some cases, it will remove the photo all together and leave a thumbnail. The next time you view the photo, it will download it again. I've only seen this happen on old photos, or when re-installing iOS fresh.

When you make this change you will notice that the Camera Roll is no longer available. Instead it is replaced with All Photos. That is because the photos are no longer stored on-device. Now they are stored in a shared library with multiple iOS devices, which requires more than what the original Camera Roll provided.

Now that we have iCloud Photo Library setup on your iOS devices to upload all of their photos and movies, we need to setup OS X. The Mac works much the same as iOS. You open up the Photos app (requires Yosemite or newer) and open up it's preferences. The only difference is that we want to tell the app to download all of the originals onto our Mac's hard-drive. This way, we always have our originals locally and aren't depending on the cloud keeping them forever.

Now we have all of our photos (and videos!) synchronizing wireless between our iOS devices and the Mac.

iCloud Photo Library to Lightroom

So we now have our photos and videos locally on our internal or external hard-drives. What do we do with them then? We export them from iCloud Photo Library and move them into Lightroom! I use Lightroom to handle all of my keywording and orginization of my photos and videos. The new OS X based Photos app fully supports keywording and organizing into albums and folders. It does have some issues, like not being able to move an album in and out of folders once the album is created. Keywords will sync across all of your devices, but the folder/album organization will not. Albums are only synced to iOS in a flattened layout, with no nested folders. An example of this on iOS looks like the following screenshot, where all of the years are flattened out.

As a result of this, I will continue to use Lightroom for my photo organization. I keep Lightroom as my source of truth, where all of my originals will always live. In Lightroom, they are organized like this:

So how do I get my photos from the OS X Photos app into Lightroom? I use a Smart Album in the Photos app! With Smart Albums, I can keep track of what date I last pulled photos into my Lightroom library. There are two Smart Albums I create. One that filters all photos that I have already brought into my Lightroom library, and a second that filters all photos that I have yet to export from Photos, and import into Lightroom.

So before we create our Smart Albums, we must first actually export the photos from iCloud Photo Library and get them into Lightroom. When I first turned on iCloud Photo Library, I had to wait just a little over 24 hours while it got all of the photos synchornized across the devices and downloaded onto my Mac. Your mileage will vary, depending on how many you have. You can see in the Photos preferences picture above, that it tells me my Photos library was just updated. That means I am in sync and nothing is outstanding. During a first time sync, it will count down how many photos and/or videos it has remaining to download onto your Mac.

So, if your sync is done, select the Photos library

Next you will need to select every photo in that library. You can do this with Command+A or using the Edit menu.

Lastly, we will export the selected photos from the Photos app, so that we can import them into Lightroom. Note that exporting will not remove them from iCloud Photo Library. It will instead create a copy of every photo and place the copy in the desired export folder. This is ideal, so that we can continue to view the photos and videos on our devices without any issue.

You can set some settings on your photos prior to the export taking place. I typically use the following settings for my exports.

Clicking the Export button will prompt you with where you want to store the files. You can store them anywhere for now. When we do the import into Lightroom, the Lightroom application will move them into their final resting place within your Lightroom library. So just pick a temporary location for now.

Now that the photos and videos have been exported, we need to get them into Lightroom. We can do that by opening Lightroom and doing an import.

On the left-hand side of the Lightroom import screen is the source of the files you want to bring into Lightroom. You will navigate to your temporary folder that holds your exported iCloud Photos. On the right-hand side you will select where your Lightroom library is, this will be their permanent storage location.

Now your photos are all set in Lightroom. You can go ahead and organize them, keyword them and edit them as you want. Next, make a back up of your Lightroom library! It's always good to keep a back up of your photos elsewhere once you have them all imported. Lightroom handles backups automatically for you when ever you close the app.

Once you are done organizing and keywording all of your photos and videos, go ahead and select them all in iCloud Photo Library and delete them. Once they are deleted, you will drag and drop all of the photos from Lightroom, into iCloud Photo Library. This will sync all of the keywords, meta-data, and edits made to your photos on to all of your devices.

Now you have your photos in sync across all devices. The first time you do this, it will take a bit. If you do this every couple of weeks, it won't be to much of a hastle. Let's cover how to do that now, using the Smart Albums I mentioned earlier!

Smart Albums

We will create a new Smart Album that will keep track of what photos and videos you have already imported and processed into Lightroom, and brought back into the iCloud Photo Library. A second Smart Album keeps track of what photos and videos are in your iCloud Photo Library, but not in Lightroom. These are going to be handled simply by tracking dates.

The first smart album is called "Importing - Pending", meaning these photos and videos are pending the import process into our master Lightroom library. You could call it something else, like "Need exporting" if you want.

This Smart Album will show me all of the photos and videos I need to export from iCloud Photo Library, and import into Lightroom. The nice part of using a Smart Album like this, is that now I know all of the photos I need to delete, and re-upload once I am done organizing my stuff in Lightroom. I can open this Smart Album, select all of the photos and videos, delete them, then drag and drop my stuff from Lightroom back into iCloud Photo Library. I set the date to be the date that I exported all of those photos.

The next Smart Album is what we will use to track what has already been imported into our Lightroom library. This will be called "Importing - Processed" and will have a date set to the day of the day you exported your photos. There might be some overlap when you next do some importing, but that's alright. Lightroom does a good job of detecting duplicate photos and show which ones it thinks are duplicates and won't import. Worse case, you have to review each photo for that one day. That's typically not many photos and videos and is worth the extra step to make sure you don't miss any.

Another option would be to add a keyword to every photo in your Lightroom library with a Lightroom keyword. Then you can create Smart Albums that look for that keyword. This way, new photos will lack it and can be added. The only downside is that keywords can't be applied to videos, which is why I chose not to go that route.

Now you have everything you need to keep both Photos and videos in sync across every single device, while maintaining the original, full-resolution, files on your local computer. By adding keywords correctly, you can do full searches in iOS and find photos really fast. As an example, I performed a search in my iCloud Photo Library (with nearly 9,000 files) on iOS for photos of an old car I had 8 years ago. The search results came back instantaneously.

Proper keywording of your photos in Lightroom (or in iCloud Photo Library if you don't want to use Lightroom) can really make it easy to find your photos.

Why Lightroom?

I used Lightroom in this example just because it is the photo suite that I use. The entire section based on Lightroom can be applied to the old iPhoto app or Aperture. If you want to keep your photos in iCloud Photo Library, you can do all the keywording within that app and not both with the whole export/import process. Instead just build smart albums that have look for a "processed" keyword, and add that keyword to every photo you process in iCloud Photo Library directly.

AppleTv

Sadly, the new iCloud Photo Library is not supported by the current generation of AppleTV. I'm not sure why, other than the fact that storage on the device might be an issue (iCloud Photo Library includes videos, where Photostream does not). I'm not sure, so for the time being when you have visitors, you'll need to Airplay your photos to your TV from an iOS device or from your Mac.

Another option could be to use the Flickr app. I have not used the app personally, so I'm not sure what options it has. The app does exist on AppleTV and you can upload every photo in your Lightroom library up to Flickr for viewing on the AppleTV.

I'm hoping I'll be able to revise this post once the new AppleTV is released, with a guide on viewing the iCloud Photo Library on it. There's no official word yet that it will be supported.

Mud Designers new home on GitHub

I had previously been maintaining two different repositories for the Mud Designer. The first was on my personal GitHub account and the other was on Codeplex.

A few weeks ago I created a new team on Git Hub called Mud Designer. There are several repositories under the team, such as the Mud Engine, the new cross-platform server stuff and the documentation source for the real help docs.

In order to help people navigate the repositories, I have setup a Home repository. For all things Mud Designer, the Mud Designer Home should be your starting place.

Some of the repositories have source in a branch other than Master right now as its being developed. So be sure to check all of the branches out when exploring the code.

Project milestones have been defined on the Mud Engine repository and the server repository. Over the next couple of weeks I will be creating backlog work items in the issue tracker, so people can get an idea Of how far along the project is in each milestone.

The previous repositories found on Codeplex and under my account will be deprecated going forward.