Gradle aws plugin written in kotlin

Intro

In the previous article we discussed creation of aws lambda function written in kotlin so we can benefit from the next advantages:

aws lambda is super scalable
it is really cheap
kotlin syntax helps to build robust implementation without overhead
aws java api is nicer to use because of the static typing which makes development easier and more productive

I also mentioned aws gradle plugin that was used to package and deploy lambda. The straightforward approach for deployment is to use aws cli tool so you need to install it, you need to have python 3 and you have to know commands. That’s why usage of the plugin which simplifies this process seems like a good idea. The plugin helps us to abstract from aws cli tool, so we don’t need to know the commands and their sequence to deploy application - plugin does it instead of us.

In the current article we will discuss this gradle plugin in details.

Aws lambda deployment process

The whole deployment process of lambda function written in java can be divided in several steps:

Fat jar file creation - copying all the dependencies as archives inside our final jar file. It is crucial for java lambda function because for now aws lambda function environment does not have way to download dependencies by itself and they have to be provided packed within jar file.

NOTE: The most obvious indicator of absence of those dependencies is ‘ClassNotFoundException’ which points to the class from our dependencies.

Packaging - for jar file in order to be used by aws cloudformation it has to be uploaded to s3 bucket and the path to the file has to be inserted into the cloudformation template as a path to sources.
Deployment - when previous two steps are finished we have ‘FatJar’ and prepared cloudformation template with the source path pointing to jar file. This artifacts are utilized by deploy command which creates related cloudformation stack in your aws account. These operations will be wrapped in our gradle plugin.

Implementation of the aws gradle plugin written in kotlin

Implementation of the plugin is pretty simple and will be done in kotlin as well as gradle configuration for it, so everything will be implemented in one language.

Entire implementation of such simple plugin can be described in a few steps:

Create plugin class which contains plugin extension registration and creation of the tasks with their configuration.
Implement tasks

I consider task as something similar to rest controller - it will react to the user interaction and start all necessary processes. In our case task picks up plugin configuration and passes it to specific service. Services will just abstract hard duty functionality from the tasks.

So this is our implementation of aws plugin class:

package com.javarubberduck.awsplugin

import com.javarubberduck.awsplugin.tasks.DeployLambdaTask
import com.javarubberduck.awsplugin.tasks.FatJarTask
import com.javarubberduck.awsplugin.tasks.PackageLambdaTask
import org.gradle.api.Plugin
import org.gradle.api.Project

class AwsPlugin: Plugin<Project> {

    private val BUILD_FAT_JAR_TASK_NAME = "buildFatJar"
    private val PACKAGE_LAMBDA_TASK_NAME = "packageLambda"
    private val DEPLOY_LAMBDA_TASK_NAME = "deployLambda"

    override fun apply(target: Project) {
        createExtension(target)
        registerTasks(target)
        configurePackageLambdaTask(target)
        configureDeployLambdaTask(target)
    }

    private fun createExtension(target: Project) {
        target.extensions.add("awsPlugin", AwsPluginExtension::class.java)
    }

    private fun registerTasks(target: Project) {
        target.tasks.register(BUILD_FAT_JAR_TASK_NAME, FatJarTask::class.java)
        target.tasks.register(PACKAGE_LAMBDA_TASK_NAME, PackageLambdaTask::class.java)
        target.tasks.register(DEPLOY_LAMBDA_TASK_NAME, DeployLambdaTask::class.java)
    }

    private fun configurePackageLambdaTask(target: Project) {
        val packageLambdaTask = target.tasks.getByName(PACKAGE_LAMBDA_TASK_NAME)
        packageLambdaTask.dependsOn(BUILD_FAT_JAR_TASK_NAME)
        packageLambdaTask.mustRunAfter(BUILD_FAT_JAR_TASK_NAME)
    }

    private fun configureDeployLambdaTask(target: Project) {
        val deployLambdaTask = target.tasks.getByName(DEPLOY_LAMBDA_TASK_NAME)
        deployLambdaTask.dependsOn(PACKAGE_LAMBDA_TASK_NAME)
        deployLambdaTask.mustRunAfter(PACKAGE_LAMBDA_TASK_NAME)
    }
}

As you can see there are several operations:

Creation of the extension - this extension allows to configure plugin in gradle build file.
Tasks registration - which enables tasks to be used through gradle
Tasks configuration - this part is needed to organize lifecycle: which tasks are dependent and in which order they have to be executed.

Next FatJarTask is pretty simple, all we have to do is to extend Jar task from java plugin and configure this task to copy all dependencies into jar file.

package com.javarubberduck.awsplugin.tasks

import org.gradle.jvm.tasks.Jar

open class FatJarTask : Jar() {
    init {
        from(project.configurations.getByName("runtimeClasspath").map { config ->
            if (config.isDirectory) config as Any else project.zipTree(config) as Any })
        with(project.tasks.getByName("jar") as Jar)
    }
}

NOTE: in this case configuration has to be done in the constructor, otherwise during invocation of this task an error will be thrown which indicates that we cannot touch dependencies anymore.

Package and Deploy tasks utilize aws java sdk 2.0. Using this sdk we can do packaging of our source code into s3 bucket and deploy the whole stack to aws account. All we need to do is to extend DefaultTask, create aws client and use it.

In PackageLambda task we need s3 client to create (if it is needed) s3 bucket, put jar file into it.

package com.javarubberduck.awsplugin.services

import com.javarubberduck.awsplugin.AwsPluginExtension
import com.javarubberduck.awsplugin.services.TemplateUtil.Companion.extractJarPath
import org.gradle.api.GradleScriptException
import org.gradle.api.InvalidUserDataException
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider
import software.amazon.awssdk.core.sync.RequestBody
import software.amazon.awssdk.regions.Region
import software.amazon.awssdk.services.s3.S3Client
import software.amazon.awssdk.services.s3.model.BucketAlreadyOwnedByYouException
import software.amazon.awssdk.services.s3.model.PutObjectRequest
import software.amazon.awssdk.services.s3.model.S3Exception
import java.io.File

class S3BucketService {
    private lateinit var extension: AwsPluginExtension
    private lateinit var s3Client: S3Client

    fun upload(extension: AwsPluginExtension) {
        s3Client = getS3Client()
        this.extension = extension
        createBucket()
        uploadJarFileToS3(extractJarPath(extension.templatePath))
    }

    private fun createBucket() {
        try {
            println("Create bucket operation is started\nbucket_name: ${extension.s3BucketName}\nprofile: ${extension.profileName}")
            s3Client.use { client ->
                client.createBucket { requestBuilder ->
                    requestBuilder.bucket(extension.s3BucketName)
                }
            }
        } catch (e: BucketAlreadyOwnedByYouException) {
            println("${extension.s3BucketName} bucket is already owned by you. No need to create it")
        } catch (e: S3Exception) {
            throw GradleScriptException(e.localizedMessage, e)
        } catch (e: Throwable) {
            throw InvalidUserDataException("Create bucket operation failed")
        }
    }

    private fun uploadJarFileToS3(jarPath: String) {
        val requestBody = RequestBody.fromFile(File(jarPath))
        val request = PutObjectRequest.builder()
                .bucket(extension.s3BucketName).key("${extension.stackName}.jar").build()
        s3Client.putObject(request, requestBody)
    }

    private fun getS3Client(): S3Client {
        return S3Client.builder()
                .region(Region.of(extension.region))
                .credentialsProvider(ProfileCredentialsProvider.builder()
                        .profileName(extension.profileName)
                        .build())
                .build()
    }
}

In DeployLambda task we are using cloudformation client to validate template, to check does stack with specified name already exist or not and to create/update stack in aws.

package com.javarubberduck.awsplugin.services

import com.javarubberduck.awsplugin.AwsPluginExtension
import org.gradle.api.InvalidUserDataException
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider
import software.amazon.awssdk.regions.Region
import software.amazon.awssdk.services.cloudformation.CloudFormationClient
import software.amazon.awssdk.services.cloudformation.model.Capability
import software.amazon.awssdk.services.cloudformation.model.CloudFormationException
import software.amazon.awssdk.services.cloudformation.model.StackStatus


class CloudFormationService {

    fun deployTemplate(extension: AwsPluginExtension) {
        val cloudFormation = CloudFormationClient.builder()
                .region(Region.of(extension.region))
                .credentialsProvider(ProfileCredentialsProvider.builder()
                        .profileName(extension.profileName)
                        .build())
                .build()

        val template = TemplateUtil.readFileContent(extension.templatePath)

        cloudFormation.use { client ->
            validateTemplate(client, template)
            val stackExists = client.listStacks {
                it.stackStatusFilters(StackStatus.CREATE_COMPLETE)
            }.stackSummaries().any { it.stackName() == extension.stackName }

            if (stackExists) {
                println("Stack already exists $stackExists and will be updated")
                client.updateStack {
                    it.templateBody(TemplateUtil.replaceJarPathWithUri(extension.s3BucketName, extension.stackName, template))
                    it.capabilities(Capability.CAPABILITY_IAM, Capability.CAPABILITY_AUTO_EXPAND)
                    it.stackName(extension.stackName)
                } as Any
            } else {
                println("Creation of the new stack is started")
                client.createStack {
                    it.templateBody(TemplateUtil.replaceJarPathWithUri(extension.s3BucketName, extension.stackName, template))
                    it.capabilities(Capability.CAPABILITY_IAM, Capability.CAPABILITY_AUTO_EXPAND)
                    it.stackName(extension.stackName)
                } as Any
            }
        }
    }

    private fun validateTemplate(client: CloudFormationClient, template: String) {
        println("Template validation started")
        try {
            client.validateTemplate {
                it.templateBody(template)
            }
        } catch (e: CloudFormationException) {
            println(e.localizedMessage)
            throw InvalidUserDataException("Cloudformation script is not valid")
        }
        println("Template is valid")
    }
}

So as you can see there is not a lot of things to be done to utilize aws java sdk and it can save us time during development of serverless applications.

We are done with the implementation so let’s look at gradle build file.

Configuration

import org.jetbrains.kotlin.gradle.tasks.KotlinCompile

group = "com.javarubberduck"
version = "0.1-SNAPSHOT"

plugins {
    kotlin("jvm") version "1.3.61"
    `maven-publish`
    `java-gradle-plugin`
}

dependencies {
    implementation(kotlin("stdlib-jdk8"))
    implementation(platform("software.amazon.awssdk:bom:2.9.18"))
    implementation("software.amazon.awssdk:s3")
    implementation("software.amazon.awssdk:cloudformation")
    testImplementation("org.junit.jupiter:junit-jupiter-api:5.5.2")
    testRuntime("org.junit.jupiter:junit-jupiter-engine:5.5.2")
    testImplementation("org.jetbrains.kotlin:kotlin-reflect")
}
repositories {
    mavenCentral()
    jcenter()
}

tasks.test {
    useJUnitPlatform()
}

val compileKotlin: KotlinCompile by tasks
compileKotlin.kotlinOptions {
    jvmTarget = "1.8"
}
val compileTestKotlin: KotlinCompile by tasks
compileTestKotlin.kotlinOptions {
    jvmTarget = "1.8"
}

gradlePlugin {
    plugins {
        create("awsPlugin") {
            id = "com.javarubberduck.awsplugin"
            implementationClass = "com.javarubberduck.awsplugin.AwsPlugin"
        }
    }
}

As you can see there is gradle plugin for plugins. It simplifies plugin creation process:

automatically applies the Java Library plugin
adds the gradleApi dependency to the configuration
enables auto validation of plugin metadata during jar task execution
adds testKit dependencies

Maven publish plugin is important for us because we don’t want to publish this aws plugin in the internet (at least for now) but still it has to be possible to use it in other projects. To do this all we need is:

properly configure maven publish plugin
build and publish aws plugin locally
import aws plugin in other projects

Configuration of the plugin can be found on the bottom of the build script.

Worth to mention this dependency "software.amazon.awssdk:bom:2.9.18" which is bill of materials file of aws java sdk - it allows us to specify other aws dependencies without versions and they will always stay compliant to each other.

Usage example

To use this plugin all we need is to write this lines of code in our build.gradle file:

import com.javarubberduck.awsplugin.AwsPlugin
import com.javarubberduck.awsplugin.AwsPluginExtension

apply<AwsPlugin>()

configure<AwsPluginExtension> {
    s3BucketName = project.name
    profileName = "personal"
    templatePath = "./resources/sam-template.yml"
    stackName = "kotlin-lambda"
    region = "eu-west-1"
}

You can find the full code of this plugin here.

Twitter Facebook LinkedIn

Gradle aws plugin written in kotlin

Intro

Aws lambda deployment process

Implementation of the aws gradle plugin written in kotlin

Configuration

Usage example

You May Also Enjoy

Typescript in AWS lambdas using built in sam functionality

Typescript in AWS lambdas

Aws lambda written in kotlin

The list of videos that can help us to stay motivated