Google Cloud Platform Blog

Introducing Grafeas: An open-source API to audit and govern your software supply chain

Thursday, October 12, 2017

By Stephen Elliott, Product Manager, Developer Platforms and Jianing Guo, Product Manager, Container Security “Shopify was looking for a comprehensive way to track and govern all the containers we ship to production. We ship over 6,000 builds every weekday and maintain a registry with over 330,000 container images. By integrating Grafeas and Kritis into our Kubernetes pipeline, we are now able to automatically store vulnerability and build information about every container image that we create and strictly enforce a built-by-Shopify policy: our Kubernetes clusters only run images signed by our builder. Grafeas and Kritis actually help us achieve better security while letting developers focus on their code. We look forward to more companies integrating with the Grafeas and Kritis projects.” — Jonathan Pulsifer, Senior Security Engineer at Shopify. (Read more in Shopify’s blog post.) The challenge of governance at scale

Growing, fragmented toolsets: As an organization grows in size and scope, it tends to use more development languages and tools, making it difficult to maintain visibility and control of its development lifecycle.

Open-source software adoption: While open-source software makes developers more productive, it also complicates auditing and governance.

Decentralization and continuous delivery: The move to decentralize engineering and ship software continuously (e.g., “push on green”) accelerates development velocity, but makes it difficult to follow best practices and standards.

Hybrid cloud deployments: Enterprises increasingly use a mix of on-premises, private and public cloud clusters to get the best of each world, but find it hard to maintain 360-degree visibility into operations across such diverse environments.

Microservice architectures: As organizations break down large systems into container-based microservices, it becomes harder to track all the pieces.

The Grafeas approach

Using immutable infrastructure (e.g., containers) to establish preventative security postures against persistent advanced threats

Building security controls into the software supply chain, based on comprehensive component metadata and security attestations, to protect production deployments

Keeping the system flexible and ensuring interoperability of developer tools around common specifications and open-source software

Universal coverage: Grafeas stores structured metadata against the software component’s unique identifier (e.g., container image digest), so you don’t have to co-locate it with the component’s registry, and so it can store metadata about components from many different repositories.

Hybrid cloud-friendly: Just as you can use JFrog Artifactory as the central, universal component repository across hybrid cloud deployments, you can use the Grafeas API as a central, universal metadata store.

Pluggable: Grafeas makes it easy to add new metadata producers and consumers (for example, if you decide to add or change security scanners, add new build systems, etc.)

Structured: Structured metadata schemas for common metadata types (e.g., vulnerability, build, attestation and package index metadata) let you add new metadata types and providers, and the tools that depend on Grafeas can immediately understand those new sources.

Strong access controls: Grafeas allows you to carefully control access for multiple metadata producers and consumers.

Rich query-ability: With Grafeas, you can easily query all metadata across all of your components so you don’t have to parse monolithic reports on each component.

Defragmenting and centralizing metadata

(click to enlarge)

JFrog is implementing Grafeas in the JFrog Xray API and will support hybrid cloud workflows that require metadata in one environment (e.g., on-premises in Xray) to be used elsewhere (e.g., on Google Cloud Platform). Read more on JFrog’s blog.

Red Hat is planning on enhancing the security features and automation of Red Hat Enterprise Linux container technologies in OpenShift with Grafeas. Read more on Red Hat’s blog.

IBM plans to deliver Grafeas and Kristis as part of the IBM Container Service on IBM Cloud, and to integrate our Vulnerability Advisor and DevOps tools with the Grafeas API. Read more on IBM’s blog.

Black Duck is collaborating with Google to implement the Google artifact metadata API implementation of Grafeas, to bring improved enterprise-grade open-source security to Google Container Registry and Google Container Engine. Read more on Black Duck’s blog.

Twistlock will integrate with Grafeas to publish detailed vulnerability and compliance data directly into orchestration tooling, giving customers more insight and confidence about their container operations. Read more on Twistlock’s blog.

Aqua Security will integrate with Grafeas to publish vulnerabilities and violations, and to enforce runtime security policies based on component metadata information. Read more on Aqua’s blog.

CoreOS is exploring integrations between Grafeas and Tectonic, its enterprise Kubernetes platform, allowing it to extend its image security scanning and application lifecycle governance capabilities.

JFrog’s Xray implementation of Grafeas API

A Google artifact metadata API implementation of Grafeas, together with Google Container Registry vulnerability scanning

Bi-directional metadata sync between JFrog Xray and the Google artifact metadata API

Black Duck integration with Grafeas and the Google artifact metadata API

Join us!

Try Grafeas now and join the GitHub project: https://github.com/grafeas

Attend Shopify’s talks at Google Cloud Summit in Toronto on 10/17 and KubeCon in December

Fill out this form to learn more about upcoming releases or talk to us about integrations

See grafeas.io for documentation and examples

4 ways you can deploy an ASP.NET Core app to GCP

Wednesday, October 11, 2017

By Ivan Naranjo, C# Team Lead running .NET Core apps on App Engine flexible environmentApp Engine flexible environmentContainer EngineGoogle Cloud Platform

Deploy from Visual Studio directly using the Cloud Tools for Visual Studio extension

Deploy a Framework Dependent Deployment bundle with "dotnet publish"

Deploy to App Engine flexible environment with a custom Dockerfile

Deploy to Container Engine with a custom Dockerfile

Build artifacts, such as .dlls, that are the result of publishing your app and which include all the necessary dependencies to be able to run

In the case of App Engine, an app.yaml file that defines the deployment, and that sits at the root of the app’s deployment files

Method 1: Deploy from Visual Studio Cloud Tools for Visual Studio

Method 2: Deploy a Framework Dependent Deployment bundle Framework Dependent Deployment bundle<ItemGroup> <None Include="app.yaml" CopyToOutputDirectory="PreserveNewest" /> </ItemGroup>runtime: aspnetcore env: flexadditional app.yaml settings

From the startup project’s directory, run “dotnet publish” with the configuration of your choice, for example:

dotnet publish -c Release

This publishes the release build to the directory “bin\<configuration>\netcore<version>\publish” with the app’s deployment files, including app.yaml if you included it in .csproj.

Deploy the app to App Engine flexible by running:

gcloud app deploy 
bin\<configuration>\netcore<version>\publish\app.yaml

This deploys the app to App Engine for you, and gcloud takes care of all of the complexities of wrapping your app into a Docker image.

Method 3: Deploy to App Engine with a custom Dockerfile runtime: custom env: flex

Creating a Dockerfile for a published app

If you want to keep going down the published app route, you can specify a Dockerfile to build the app’s image based on a published directory. The Dockerfile looks like this:

FROM gcr.io/google-appengine/aspnetcore:2.0
ADD ./ /app
ENV ASPNETCORE_URLS=http://*:${PORT}
WORKDIR /app
ENTRYPOINT [ "dotnet", "MainProject.dll" ]

Notice how this Dockerfile example uses the “gcr.io/google-appengine/aspnetcore:2.0” as its base image. We've created a set of Docker images that are optimized for running ASP.NET Core apps in App Engine and Container Engine, and we highly recommend that you use them for your custom Dockerfiles. These are the same images we use when we generate Dockerfiles for you during deployment. Be sure to change it to refer to the correct .NET Core runtime version for your app.

To ensure that the Dockerfile is in the published directory, add it to your .csproj so it's published when the app is published. Assuming that the Dockerfile is in the root of the startup project, on the same level as .csproj and app.yaml, add the following snippet to .csproj:

<ItemGroup>
  <None Include="app.yaml" CopyToOutputDirectory="PreserveNewest" />
  <None Include="Dockerfile" CopyToOutputDirectory="PreserveNewest" />
</ItemGroup>

With this change, whenever you run “dotnet publish” both files are copied to the published directory. To deploy, just follow the same steps as before:

dotnet publish -c Release

gcloud app deploy 
bin\<configuration>\netcore<version>\publish\app.yaml

Creating a Dockerfile that compiles and publishes the app
If you can easily build your ASP.NET Core app on Linux, you should consider using a multi-stage Dockerfile that performs the restore and publish steps during the build process, before building the final app’s image. This makes it a bit more convenient to deploy the app, as the build is done during deployment.

Here’s what that Dockerfile looks like:

# First let’s build the app and publish it.
FROM gcr.io/cloud-builders/csharp/dotnet AS builder
COPY . /src
WORKDIR /src
RUN dotnet restore --packages /packages
RUN dotnet publish -c Release -o /published

# Now let's build the app's image.
FROM gcr.io/google-appengine/aspnetcore:2.0
COPY --from=builder /published /app
ENV ASPNETCORE_URLS=http://*:${PORT}
WORKDIR /app
ENTRYPOINT [ "dotnet", "multistage-2.0.dll" ]

This Dockerfile uses the “gcr.io/cloud-builders/csharp/dotnet” builder image that wraps the .NET SDKs for all supported versions of .NET Core.

The main advantage of this method is that you can put app.yaml and the Dockerfile in the root of your project, next to .csproj. Then, to deploy the app, simply run the following command:

gcloud app deploy app.yaml

This uploads your app sources to Cloud Builder where the build will take place. The resulting build artifacts will then be used to produce the final app image, which will be deployed to App Engine. You can also run tests during the build process, making this a complete CI/CD solution.
Method 4: Deploy to Container Engine Container Engine— — Building the app’s Docker imageCloud Container RegistryCloud Container Builderkubectl run myservice --image=gcr.io/<your project id>/<app name> --port=8080This command builds a Docker image called gcr.io/$PROJECT_ID/<app name> where $PROJECT_ID is automatically substituted for your GCP project ID and <app name> is the name of your app.
Deploying the image to a Container Engine cluster kubectlgcloud components install kubectlkubectlgcloud container clusters get-credentialskubectl run myservice --image=gcr.io/<your project id>/<app name> --port=8080--replicas=nnNote: Here, we assume that your Docker containers export port 8080, the default for App Engine flexible environment, and that you expose services from port 80. To change these defaults, read about how to configure HTTPS support for public Kubernetes services.kubectl expose deployment myservice --port=80 --target-port=8080 --type=LoadBalancer In conclusionour .NET pagehttps://github.com/GoogleCloudPlatform/aspnet-dockeraspnet-docker repo

GCP adds support for multiple network interfaces

Tuesday, October 10, 2017

By Neal Mueller, Product Marketing Lead and Ines Envid, Product Manager Virtual Private Cloud

Connect virtual network and security appliances

Isolate public-facing services from an internal network and its services

Separate management, control, storage and data plane networks

Create an inexpensive fault-tolerant solution

[Editor’s note: If you’d like to build your own architectural diagrams such as this, check out these sample diagrams and our icon library.]

"We have been working closely with Google Cloud on design and use cases for this capability. The multiple network interface VM will enable Palo Alto Networks to provide the same enterprise-grade security that customers are used to in their private data centers. Customers will be able to inspect not just the traffic coming into GCP, but also the East-West traffic between their GCP projects and across VPCs." — Adam Geller, VP, Product Management for Virtualization and Cloud at Palo Alto Networks

"We are delighted to have worked with Google to demonstrate how NETSCOUT’s packet-based application assurance can be extended to multiple interface GCP compute instances. This will allow GCP customers to leverage the benefits of multiple network interfaces, while minimizing the disruption of cloud migration and hybrid cloud deployments through the proactive identification of issues impacting user experience, operational efficiency and productivity." — Paul Barrett, CTO for Enterprise Business Operations

documentationjoin the partner community

Building reliable deployments with Spinnaker, Container Engine and Container Builder

Monday, October 9, 2017

By Vic Iglesias, Cloud Solutions Architect
primitives to help you deployContainer EngineContainer BuilderSpinnaker

continuous delivery pipeline from scratch using Container Builder and Spinnaker

Spinnaker solutionSpinnaker’s pipeline stages

Customizing Stackdriver Logs for Container Engine with Fluentd

Monday, October 9, 2017

By John La Barge, Cloud Solutions Architect
Google Cloud PlatformContainer EngineStackdriverFluentdthis tutorial

Using Stackdriver Logging for dedicated game server instances: new tutorial

Friday, October 6, 2017

Joseph Holley, Cloud Solutions Architect
tutorialStackdriver Logging@gcpjoe

Now shipping: Compute Engine machine types with up to 96 vCPUs and 624GB of memory

Thursday, October 5, 2017

By Hanan Youssef and Scott Van Woudenberg, Product Managers, Compute Engine
Intel Xeon Scalable processors¹predefined machine types

Standard: 96 vCPUs and 360 GB of memory

High-CPU: 96 vCPUs and 86.4 GB of memory

High-Memory: 96 vCPUs and 624 GB of memory

custom machine typesextended memory

GCP Consolecreating new virtual machines using the gcloud command line toolearly testing group for new machine types
1 Based on comparing Intel Xeon Scalable Processor codenamed "Skylake" versus previous generation Intel Xeon processor codenamed "Broadwell." 20% based on SpecINT. 82% based on on High Performance Linpack for 4 node cluster with AVX512. Performance improvements include improvements from use of Math Kernel Library and Intel AVX512. Performance tests are measured using specific computer systems, components, software, operations and functions, and may have been optimized for performance only on Intel microprocessors. Any change to any of those factors may cause the results to vary. You should consult other information to assist you in fully evaluating your contemplated purchases. For more information go to http://www.intel.com/benchmarks^↩