LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Bart Van Assche <bvanassche@acm.org>
To: longli@linuxonhyperv.com, linux-block@vger.kernel.org,
	linux-kernel@vger.kernel.org, linux-hyperv@vger.kernel.org
Cc: Long Li <longli@microsoft.com>, Jonathan Corbet <corbet@lwn.net>,
	"K. Y. Srinivasan" <kys@microsoft.com>,
	Haiyang Zhang <haiyangz@microsoft.com>,
	Stephen Hemminger <sthemmin@microsoft.com>,
	Wei Liu <wei.liu@kernel.org>, Dexuan Cui <decui@microsoft.com>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Bjorn Andersson <bjorn.andersson@linaro.org>,
	Hans de Goede <hdegoede@redhat.com>,
	Dan Williams <dan.j.williams@intel.com>,
	Maximilian Luz <luzmaximilian@gmail.com>,
	Mike Rapoport <rppt@kernel.org>,
	Ben Widawsky <ben.widawsky@intel.com>,
	Jiri Slaby <jirislaby@kernel.org>,
	Andra Paraschiv <andraprs@amazon.com>,
	Siddharth Gupta <sidgup@codeaurora.org>,
	Hannes Reinecke <hare@suse.de>
Subject: Re: [Patch v5 0/3] Introduce a driver to support host accelerated access to Microsoft Azure Blob for Azure VM
Date: Thu, 5 Aug 2021 10:09:08 -0700	[thread overview]
Message-ID: <e249d88b-6ca2-623f-6f6e-9547e2b36f1f@acm.org> (raw)
In-Reply-To: <1628146812-29798-1-git-send-email-longli@linuxonhyperv.com>

On 8/5/21 12:00 AM, longli@linuxonhyperv.com wrote:
> From: Long Li <longli@microsoft.com>
> 
> Azure Blob storage [1] is Microsoft's object storage solution for the
> cloud. Users or client applications can access objects in Blob storage via
> HTTP, from anywhere in the world. Objects in Blob storage are accessible
> via the Azure Storage REST API, Azure PowerShell, Azure CLI, or an Azure
> Storage client library. The Blob storage interface is not designed to be a
> POSIX compliant interface.
> 
> Problem: When a client accesses Blob storage via HTTP, it must go through
> the Blob storage boundary of Azure and get to the storage server through
> multiple servers. This is also true for an Azure VM.
> 
> Solution: For an Azure VM, the Blob storage access can be accelerated by
> having Azure host execute the Blob storage requests to the backend storage
> server directly.
> 
> This driver implements a VSC (Virtual Service Client) for accelerating Blob
> storage access for an Azure VM by communicating with a VSP (Virtual Service
> Provider) on the Azure host. Instead of using HTTP to access the Blob
> storage, an Azure VM passes the Blob storage request to the VSP on the
> Azure host. The Azure host uses its native network to perform Blob storage
> requests to the backend server directly.
> 
> This driver doesn’t implement Blob storage APIs. It acts as a fast channel
> to pass user-mode Blob storage requests to the Azure host. The user-mode
> program using this driver implements Blob storage APIs and packages the
> Blob storage request as structured data to VSC. The request data is modeled
> as three user provided buffers (request, response and data buffers), that
> are patterned on the HTTP model used by existing Azure Blob clients. The
> VSC passes those buffers to VSP for Blob storage requests.
> 
> The driver optimizes Blob storage access for an Azure VM in two ways:
> 
> 1. The Blob storage requests are performed by the Azure host to the Azure
> Blob backend storage server directly.
> 
> 2. It allows the Azure host to use transport technologies (e.g. RDMA)
> available to the Azure host but not available to the VM, to reach to Azure
> Blob backend servers.
>   
> Test results using this driver for an Azure VM:
> 100 Blob clients running on an Azure VM, each reading 100GB Block Blobs.
> (10 TB total read data)
> With REST API over HTTP: 94.4 mins
> Using this driver: 72.5 mins
> Performance (measured in throughput) gain: 30%.
>   
> [1] https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction

Is the ioctl interface the only user space interface provided by this 
kernel driver? If so, why has this code been implemented as a kernel 
driver instead of e.g. a user space library that uses vfio to interact 
with a PCIe device? As an example, Qemu supports many different virtio 
device types.

Thanks,

Bart.


  parent reply	other threads:[~2021-08-05 17:09 UTC|newest]

Thread overview: 35+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-08-05  7:00 longli
2021-08-05  7:00 ` [Patch v5 1/3] Drivers: hv: vmbus: add support to ignore certain PCIE devices longli
2021-08-05  7:00 ` [Patch v5 2/3] Drivers: hv: add Azure Blob driver longli
2021-08-05  7:11   ` Greg Kroah-Hartman
2021-08-05 18:07     ` Long Li
2021-08-05 18:16       ` Greg Kroah-Hartman
2021-08-05 17:06   ` Bart Van Assche
2021-08-05 18:10     ` Long Li
2021-08-05 18:17     ` Greg Kroah-Hartman
2021-09-07 21:42   ` Michael Kelley
2021-08-05  7:00 ` [Patch v5 3/3] Drivers: hv: Add to maintainer for Hyper-V/Azure drivers longli
2021-08-05  7:08 ` [Patch v5 0/3] Introduce a driver to support host accelerated access to Microsoft Azure Blob for Azure VM Greg Kroah-Hartman
2021-08-05 18:27   ` Long Li
2021-08-05 18:33     ` Greg Kroah-Hartman
2021-08-05 17:09 ` Bart Van Assche [this message]
2021-08-05 18:24   ` Long Li
2021-08-05 18:34     ` Greg Kroah-Hartman
2021-08-07 18:29       ` Long Li
2021-08-08  5:14         ` Greg Kroah-Hartman
2021-08-10  3:01           ` Long Li
2021-09-22 23:55             ` Long Li
2021-09-30 22:25               ` Long Li
2021-10-01  7:36                 ` Greg Kroah-Hartman
2021-10-07 18:15                   ` Long Li
2021-10-08  5:54                     ` Greg Kroah-Hartman
2021-10-08 11:11                       ` Vitaly Kuznetsov
2021-10-08 11:19                         ` Greg Kroah-Hartman
2021-10-08 13:28                           ` Vitaly Kuznetsov
2021-10-11 17:57                             ` Long Li
2021-10-13  0:58                               ` Long Li
2021-10-13  7:03                                 ` Greg Kroah-Hartman
2021-10-11 17:55                           ` Long Li
2021-10-11 17:46                         ` Long Li
2021-10-11 17:58                           ` Greg Kroah-Hartman
2021-10-11 19:38                             ` Long Li

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=e249d88b-6ca2-623f-6f6e-9547e2b36f1f@acm.org \
    --to=bvanassche@acm.org \
    --cc=andraprs@amazon.com \
    --cc=ben.widawsky@intel.com \
    --cc=bjorn.andersson@linaro.org \
    --cc=corbet@lwn.net \
    --cc=dan.j.williams@intel.com \
    --cc=decui@microsoft.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=haiyangz@microsoft.com \
    --cc=hare@suse.de \
    --cc=hdegoede@redhat.com \
    --cc=jirislaby@kernel.org \
    --cc=kys@microsoft.com \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-hyperv@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=longli@linuxonhyperv.com \
    --cc=longli@microsoft.com \
    --cc=luzmaximilian@gmail.com \
    --cc=rppt@kernel.org \
    --cc=sidgup@codeaurora.org \
    --cc=sthemmin@microsoft.com \
    --cc=wei.liu@kernel.org \
    --subject='Re: [Patch v5 0/3] Introduce a driver to support host accelerated access to Microsoft Azure Blob for Azure VM' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).