ABP Framework-Blob Storage源码解析
目录
https://abp.io/docs/6.0/Blob-Storing
Version
6.0.3
Package
Volo.Abp.BlobStoring
Volo.Abp.BlobStoring.Aliyun
Volo.Abp.BlobStoring.Aws
Volo.Abp.BlobStoring.Azure
Volo.Abp.BlobStoring.FileSystem
Volo.Abp.BlobStoring.Minio
//独立模块
Volo.Abp.BlobStoring.Database.*
BlobProvider
当有一个Blob需要存储到存储介质上,因为存储介质各有不同,调用每一个存储介质的方式有所差异,ABP在各式各样的存储介质上抽象一层,提供基础方法来确保对于Blob的常规操作,如存储,查询,删除,判断是否存在等方法。代码简要如下
public interface IBlobContainer
{
Task SaveAsync(string name, Stream stream, bool overrideExisting = false);
Task<bool> DeleteAsync(string name);
Task<bool> ExistsAsync(string name);
Task<Stream> GetAsync(string name);
Task<Stream> GetOrNullAsync(string name);
}
同时提供了Azure/Aws/Aliyun/File/Db/Minio等且支持扩展的Provider,来隔离实际的调用方式。
在不同的存储介质下,调用不同的包,此处以Minio为例,在MinioBlobProvider(Volo.Abp.BlobStoring.Minio包)中,依赖Minio包,将Blob移交到Minio的参数下。代码简要如下
public class MinioBlobProvider : BlobProviderBase, ITransientDependency
{
//...
public override async Task SaveAsync(BlobProviderSaveArgs args)
{
var blobName = MinioBlobNameCalculator.Calculate(args);
// 生成MinioClient
var client = GetMinioClient(args);
var containerName = GetContainerName(args);
//...
// 上传Blob到Minio中
await client.PutObjectAsync(containerName, blobName, args.BlobStream, args.BlobStream.Length);
}
public override async Task<bool> DeleteAsync(BlobProviderDeleteArgs args)
{
var blobName = MinioBlobNameCalculator.Calculate(args);
// 生成MinioClient
var client = GetMinioClient(args);
var containerName = GetContainerName(args);
// 请求Minio删除文件
if (await BlobExistsAsync(client, containerName, blobName))
{
await client.RemoveObjectAsync(containerName, blobName);
return true;
}
return false;
}
public override async Task<bool> ExistsAsync(BlobProviderExistsArgs args)
{
var blobName = MinioBlobNameCalculator.Calculate(args);
// 生成MinioClient
var client = GetMinioClient(args);
var containerName = GetContainerName(args);
// 请求Minio查询文件是否存在
try
{
await client.StatObjectAsync(containerName, blobName);
}
catch(Exception)
{
return false;
}
return true;
}
//...
protected virtual MinioClient GetMinioClient(BlobProviderArgs args)
{
var configuration = args.Configuration.GetMinioConfiguration();
var client = new MinioClient()
.WithEndpoint(configuration.EndPoint)
.WithCredentials(configuration.AccessKey, configuration.SecretKey);
if (configuration.WithSSL)
{
client.WithSSL();
}
return client.Build();
}
//...
/
}
对于其他Provider来讲,大致过程相同。参照同样的过程,可以扩展需要的存储介质Provider。
BlobContainerConfiguration
在生成MinioClient方法中,需要Minio参数,Azure/Aws/Aliyun等相同,都需要一些存储介质的参数设置。对于参数配置,ABP抽象了一层BlobContainerConfiguration,来管理各存储介质端预设好的参数。
public class BlobContainerConfiguration
{
public Type ProviderType { get; set; }
private readonly Dictionary<string, object> _properties;
private readonly BlobContainerConfiguration _fallbackConfiguration;
//...
public BlobContainerConfiguration(BlobContainerConfiguration fallbackConfiguration = null)
{
_fallbackConfiguration = fallbackConfiguration;
_properties = new Dictionary<string, object>();
}
//...
public object GetConfigurationOrNull(string name, object defaultValue = null)
{
return _properties.GetOrDefault(name) ??
_fallbackConfiguration?.GetConfigurationOrNull(name, defaultValue) ??
defaultValue;
}
public BlobContainerConfiguration SetConfiguration([NotNull] string name, [CanBeNull] object value)
{
//..
_properties[name] = value;
return this;
}
public BlobContainerConfiguration ClearConfiguration([NotNull] string name)
{
//...
_properties.Remove(name);
return this;
}
}
内部采用字典来保存各存储介质的参数信息。以Minio为例,当采用Minio作为存储介质时,在模块服务注册中,则配置好Minio的参数。在Volo.Abp.BlobStoring.Minio中提供了扩展方法,指定当前使用的时MinioBlobProvider。并将Minio相关参数移交到BlobContainerConfiguration中。
public static class MinioBlobContainerConfigurationExtensions
{
public static MinioBlobProviderConfiguration GetMinioConfiguration(
this BlobContainerConfiguration containerConfiguration)
{
return new MinioBlobProviderConfiguration(containerConfiguration);
}
public static BlobContainerConfiguration UseMinio(
this BlobContainerConfiguration containerConfiguration,
Action<MinioBlobProviderConfiguration> minioConfigureAction)
{
containerConfiguration.ProviderType = typeof(MinioBlobProvider);
containerConfiguration.NamingNormalizers.TryAdd<MinioBlobNamingNormalizer>();
minioConfigureAction(new MinioBlobProviderConfiguration(containerConfiguration));
return containerConfiguration;
}
}
服务注册时配置参数,最终全部存入到字典中。
public class XXXModule : AbpModule
{
public override void ConfigureServices(ServiceConfigurationContext context)
{
Configure<AbpBlobStoringOptions>(options =>
{
options.Containers.ConfigureDefault(containerConfiguration =>
{
containerConfiguration.UseMinio(minio =>
{
minio.EndPoint = "";
minio.AccessKey = "";
minio.SecretKey = "";
minio.WithSSL = false;
minio.BucketName = "";
});
});
});
}
}
此处还得提及一个在BlobContainerConfigures上再封装的类,BlobContainerConfigurations,默认在使用Container时,只有一个DefaultContainer,如果想要按照场景分类使用不同的存储存储介质,则可以额外创建具名的Container。例如
[BlobContainerName("profile-pictures")]
public class ProfilePictureContainer
{
}
如此依赖,在服务注册时可以对不同的Container使用不同的存储介质配置参数。比如默认(DefaultContainer)的使用Minio, ProfilePictureContainer使用Db。
Configure<AbpBlobStoringOptions>(options =>
{
//DefaultContainer
options.Containers.ConfigureDefault(containerConfiguration =>
{
containerConfiguration.UseMinio(minio =>
{
minio.EndPoint = "";
minio.AccessKey = "";
minio.SecretKey = "";
minio.WithSSL = false;
minio.BucketName = "";
});
});
//ProfilePictureContainer
options.Containers.Configure<ProfilePictureContainer>(container =>
{
container.UseDatabase();
});
});
BlobContainerConfigureations提供了三类注册方式
Configure()&Configure(string name),具名的Container相关配置
ConfigureDefault(),默认的DefaultContainer相关配置
ConfigureAll(),所有的Container相关配置
如上可以将多个Container及对应使用的BlobProvider映射存储到BlobContainerConfigurations中的字典中,其key对应于Container的name,value对应于BlobContainerConfiguration的配置值。
public class BlobContainerConfigurations
{
private BlobContainerConfiguration Default => GetConfiguration<DefaultContainer>();
private readonly Dictionary<string, BlobContainerConfiguration> _containers;
public BlobContainerConfigurations()
{
_containers = new Dictionary<string, BlobContainerConfiguration>
{
//Add default container
[BlobContainerNameAttribute.GetContainerName<DefaultContainer>()] = new BlobContainerConfiguration()
};
}
//...
public BlobContainerConfigurations Configure(
[NotNull] string name,
[NotNull] Action<BlobContainerConfiguration> configureAction)
{
//...
configureAction(
_containers.GetOrAdd(
name,
() => new BlobContainerConfiguration(Default)
)
);
return this;
}
public BlobContainerConfigurations ConfigureDefault(Action<BlobContainerConfiguration> configureAction)
{
configureAction(Default);
return this;
}
public BlobContainerConfigurations ConfigureAll(Action<string, BlobContainerConfiguration> configureAction)
{
foreach (var container in _containers)
{
configureAction(container.Key, container.Value);
}
return this;
}
//...
public BlobContainerConfiguration GetConfiguration([NotNull] string name)
{
//...
return _containers.GetOrDefault(name) ?? Default;
}
}
简图介绍下结构,内部是字典,存储映射关系,其key为Container名,其value中存储着详细的存储介质的参数配置。
最终都存储在AbpBlobStoringOptions中
public class AbpBlobStoringOptions
{
public BlobContainerConfigurations Containers { get; }
public AbpBlobStoringOptions()
{
Containers = new BlobContainerConfigurations();
}
}
BlobContainerConfigurationProvider
BlobContainerConfigurations和AbpBlobStoringOptions作为存储方,不直接被上层使用,在使用方和存储方之间,ABP又抽象一层Provider,来管理取的职责。
其封装的功能则是通过name(ContainerName),得到相应的BlobContainterConfiguration。在实现中,主要从Options中调用Containers上的方法,得到。
public class DefaultBlobContainerConfigurationProvider : IBlobContainerConfigurationProvider, ITransientDependency
{
protected AbpBlobStoringOptions Options { get; }
public DefaultBlobContainerConfigurationProvider(IOptions<AbpBlobStoringOptions> options)
{
Options = options.Value;
}
public virtual BlobContainerConfiguration Get(string name)
{
return Options.Containers.GetConfiguration(name);
}
}
因此,整个Provider也提供了一些扩展性,可以替换AbpBlobStoringOptions,实现自己的一套BlobContainerConfiguration存储逻辑,总归只要得到映射关系。
BlobProviderSelector
上一节中提及了BlobProvider及各类实现,以及各类实现相关配置,对于上层BlobContainer(见下小节)如何得到相应的BlobProvider,该过程存在一个选择过程,ABP封装了BlobProviderSelector,来管理BlobContainer对应的BlobProvider。
通过ContainerName得到对应的BlobProvider。
public interface IBlobProviderSelector
{
[NotNull]
IBlobProvider Get([NotNull] string containerName);
}
在默认实现中,会先根据ContainerName从BlobContainerConfigurations的字典中中找到实际存储介质参数BlobContainerConfiguration,在这其中包含了这个ContainerName实际对应着使用的BlobProvider类型。比如在之前的minio示例中,DefaultContainer对应着使用MinioBlobProvider。
containerConfiguration.ProviderType = typeof(MinioBlobProvider);
DefaultBlobProviderSelector的代码实现简要如下,同时可以看到,可以参照该设计扩展或重写其Get逻辑,从而实现符合自己业务或技术需要的获取BlobProvider逻辑。
public class DefaultBlobProviderSelector : IBlobProviderSelector, ITransientDependency
{
protected IEnumerable<IBlobProvider> BlobProviders { get; }
protected IBlobContainerConfigurationProvider ConfigurationProvider { get; }
public DefaultBlobProviderSelector(
IBlobContainerConfigurationProvider configurationProvider,
IEnumerable<IBlobProvider> blobProviders)
{
ConfigurationProvider = configurationProvider;
BlobProviders = blobProviders;
}
[NotNull]
public virtual IBlobProvider Get([NotNull] string containerName)
{
//...
var configuration = ConfigurationProvider.Get(containerName);
//...
// 根据ProviderType创建具体存储介质的BlobProvider实例
foreach (var provider in BlobProviders)
{
if (ProxyHelper.GetUnProxiedType(provider).IsAssignableTo(configuration.ProviderType))
{
return provider;
}
}
throw new AbpException(
$"Could not find the BLOB Storage provider with the type ({configuration.ProviderType.AssemblyQualifiedName}) configured for the container {containerName} and no default provider was set."
);
}
}
如此,从ContainerName到对应应该使用的存储介质BlobProvider,以及该存储介质对应的参数配置,都可以串联起来了。 ContainerName->BlobProviderSelector->BlobContainerConfigurations->BlobContainerConfiguration->BlobProvider
BlobContainer
对于上层使用方,默认的使用方式是注入IBlobContainer。
public class MyService : ITransientDependency
{
private readonly IBlobContainer _blobContainer;
public MyService(IBlobContainer blobContainer)
{
_blobContainer = blobContainer;
}
public async Task SaveBytesAsync(byte[] bytes)
{
await _blobContainer.SaveAsync("my-blob-1", bytes);
}
public async Task<byte[]> GetBytesAsync()
{
return await _blobContainer.GetAllBytesOrNullAsync("my-blob-1");
}
}
在IBlobContainer的实现中,拥有着抽象的IBlobProvider及对应的BlobContainerConfiguration配置。
public class BlobContainer : IBlobContainer
{
protected string ContainerName { get; }
protected BlobContainerConfiguration Configuration { get; }
protected IBlobProvider Provider { get; }
protected ICurrentTenant CurrentTenant { get; }
protected ICancellationTokenProvider CancellationTokenProvider { get; }
protected IServiceProvider ServiceProvider { get; }
protected IBlobNormalizeNamingService BlobNormalizeNamingService { get; }
public BlobContainer(
string containerName,
BlobContainerConfiguration configuration,
IBlobProvider provider,
ICurrentTenant currentTenant,
ICancellationTokenProvider cancellationTokenProvider,
IBlobNormalizeNamingService blobNormalizeNamingService,
IServiceProvider serviceProvider)
{
ContainerName = containerName;
Configuration = configuration;
Provider = provider;
CurrentTenant = currentTenant;
CancellationTokenProvider = cancellationTokenProvider;
BlobNormalizeNamingService = blobNormalizeNamingService;
ServiceProvider = serviceProvider;
}
//...
}
但需要注意,BlobContainer构造函数中的这些参数并不是通过DI注入,而是实例化BlobContainer的传参。
BlobContainer
在AppService中除了直接使用IBlobContainer,还可以使用IBlobContainer。两者在底层实现上是相同的,只是前者IBlobContainer在底层默认的TContainer为DefaultContainer类型,后者则是在使用时需指明TContainer具体类型。
public class ProfileAppService : ApplicationService
{
private readonly IBlobContainer<ProfilePictureContainer> _blobContainer;
public ProfileAppService(IBlobContainer<ProfilePictureContainer> blobContainer)
{
_blobContainer = blobContainer;
}
public async Task SaveProfilePictureAsync(byte[] bytes)
{
var blobName = CurrentUser.GetId().ToString();
await _blobContainer.SaveAsync(blobName, bytes);
}
public async Task<byte[]> GetProfilePictureAsync()
{
var blobName = CurrentUser.GetId().ToString();
return await _blobContainer.GetAllBytesOrNullAsync(blobName);
}
}
尽管从类图上看,IBlobContainer的直接实现为BlobContainer,但是注入IBlobContainer实现时并不是。在AbpBlobStoringModule服务注册中,IBlobContainer<>对应的注入实现为BlobContainer<>,并且IBlobContainer的注入实现注册为BlobContainer。
[DependsOn(
typeof(AbpMultiTenancyModule),
typeof(AbpThreadingModule)
)]
public class AbpBlobStoringModule : AbpModule
{
public override void ConfigureServices(ServiceConfigurationContext context)
{
context.Services.AddTransient(
typeof(IBlobContainer<>),
typeof(BlobContainer<>)
);
context.Services.AddTransient(
typeof(IBlobContainer),
serviceProvider => serviceProvider
.GetRequiredService<IBlobContainer<DefaultContainer>>()
);
}
}
深入到BlobContainer内部,其内部存在着IBlobContainer属性,而该属性值来源于工厂创建。
public class BlobContainer<TContainer> : IBlobContainer<TContainer>
where TContainer : class
{
private readonly IBlobContainer _container;
public BlobContainer(IBlobContainerFactory blobContainerFactory)
{
_container = blobContainerFactory.Create<TContainer>();
}
}
因此当调用IBlobContainer的方法时,先进入到BlobContainer中,但是又实际调用IBlobContainer的实例BlobContainer内部的方法,此处有一丝绕。 IBlobContainer->BlobContainer->BlobContainer->IBlobProvider->xxx BlobProvider
BlobContainerFactory
如上在BlobContainer内部,最终IBlobContainer的实例化来源从BlobContainerFactory中创建得到。
在默认实现中根据ContainerName通过IBlobProviderSelector取到具体的BlobProvider,传参相关参数来实例一个BlobContainer。
public class BlobContainerFactory : IBlobContainerFactory, ITransientDependency
{
protected IBlobProviderSelector ProviderSelector { get; }
protected IBlobContainerConfigurationProvider ConfigurationProvider { get; }
protected ICurrentTenant CurrentTenant { get; }
protected ICancellationTokenProvider CancellationTokenProvider { get; }
protected IServiceProvider ServiceProvider { get; }
protected IBlobNormalizeNamingService BlobNormalizeNamingService { get; }
public BlobContainerFactory(
IBlobContainerConfigurationProvider configurationProvider,
ICurrentTenant currentTenant,
ICancellationTokenProvider cancellationTokenProvider,
IBlobProviderSelector providerSelector,
IServiceProvider serviceProvider,
IBlobNormalizeNamingService blobNormalizeNamingService)
{
ConfigurationProvider = configurationProvider;
CurrentTenant = currentTenant;
CancellationTokenProvider = cancellationTokenProvider;
ProviderSelector = providerSelector;
ServiceProvider = serviceProvider;
BlobNormalizeNamingService = blobNormalizeNamingService;
}
public virtual IBlobContainer Create(string name)
{
var configuration = ConfigurationProvider.Get(name);
return new BlobContainer(
name,
configuration,
ProviderSelector.Get(name),
CurrentTenant,
CancellationTokenProvider,
BlobNormalizeNamingService,
ServiceProvider
);
}
}
对于这个name的来源,则从扩展方法中得到详解。
public static class BlobContainerFactoryExtensions
{
public static IBlobContainer Create<TContainer>(this IBlobContainerFactory blobContainerFactory)
{
var name = BlobContainerNameAttribute.GetContainerName<TContainer>();
return blobContainerFactory.Create(name);
}
}
在BlobContainer中有
public class BlobContainer<TContainer> : IBlobContainer<TContainer>
where TContainer : class
{
private readonly IBlobContainer _container;
public BlobContainer(IBlobContainerFactory blobContainerFactory)
{
_container = blobContainerFactory.Create<TContainer>();
}
//...
}
默认使用IBlobContainer时,其实际上使用的DefaultContainer在模块服务注册时写明了。
context.Services.AddTransient(
typeof(IBlobContainer),
serviceProvider => serviceProvider
.GetRequiredService<IBlobContainer<DefaultContainer>>()
);
因此,IBlobContainer最终实现和IBlobContainer是相同的,只是前者TContainer默认为DefaultContainer,后者则需要指明。
BlobNormalizerNamingService
对于Blob的存储,大多会有两个层级,一个是Container,另一个是Container下的Blob,对应于文件系统中文件夹和文件,minio中的Bucket和Object等,当借助于BlobContainer中的Save/Get/Delete方法操作时,如果传入的名字不符合各存储介质的规范要求,则会有一些错误,因此在命名方面,ABP封装了一层BlobNormalizerNamingService,来对给定命名加工处理,以规避错误。
IBlobNormalizerNamingService中定义了对Container/BlobName的规范接口。
在实现中,这个处理逻辑实则时交付给各存储介质方来管理,也就是让各个存储方来规范期望的Container/BlobName。
public class BlobNormalizeNamingService : IBlobNormalizeNamingService, ITransientDependency
{
//...
public BlobNormalizeNaming NormalizeNaming(BlobContainerConfiguration configuration, string containerName, string blobName)
{
//...
using (var scope = ServiceProvider.CreateScope())
{
// 循环调用单个存储介质方,在服务注册阶段配置好的命名标准化服务
foreach (var normalizerType in configuration.NamingNormalizers)
{
// 获得BlobNamingNormalizer实例
var normalizer = scope.ServiceProvider
.GetRequiredService(normalizerType)
.As<IBlobNamingNormalizer>();
// 命名规范化职责交给具体的存储介质方来维护
containerName = containerName.IsNullOrWhiteSpace() ? containerName : normalizer.NormalizeContainerName(containerName);
blobName = blobName.IsNullOrWhiteSpace() ? blobName : normalizer.NormalizeBlobName(blobName);
}
return new BlobNormalizeNaming(containerName, blobName);
}
}
//...
}
以Minio为例,在服务注册中,调用了UseMinio来配置BlobContainerConfiguration,在其中可以加入多个NamingNormalizers。
public static BlobContainerConfiguration UseMinio(
this BlobContainerConfiguration containerConfiguration,
Action<MinioBlobProviderConfiguration> minioConfigureAction)
{
containerConfiguration.ProviderType = typeof(MinioBlobProvider);
containerConfiguration.NamingNormalizers.TryAdd<MinioBlobNamingNormalizer>();
minioConfigureAction(new MinioBlobProviderConfiguration(containerConfiguration));
return containerConfiguration;
}
在其实际的BlobNamingNormailizer中,则是规范Container/BlobName的一些要求,仍然以Minio为例,规范Minio的ContainerName需要小写,小于63个字符等等规则,在这个规范基础上,还可以扩展或重写以适配业务或技术的需要。
public class MinioBlobNamingNormalizer : IBlobNamingNormalizer, ITransientDependency
{
/// <summary>
///https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html
/// </summary>
public virtual string NormalizeContainerName(string containerName)
{
using (CultureHelper.Use(CultureInfo.InvariantCulture))
{
// All letters in a container name must be lowercase.
containerName = containerName.ToLower();
// Container names must be from 3 through 63 characters long.
if (containerName.Length > 63)
{
containerName = containerName.Substring(0, 63);
}
// Bucket names can consist only of lowercase letters, numbers, dots (.), and hyphens (-).
containerName = Regex.Replace(containerName, "[^a-z0-9-.]", string.Empty);
// Bucket names must begin and end with a letter or number.
// Bucket names must not be formatted as an IP address (for example, 192.168.5.4).
// Bucket names can't start or end with hyphens adjacent to period
// Bucket names can't start or end with dots adjacent to period
containerName = Regex.Replace(containerName, "\\.{2,}", ".");
containerName = Regex.Replace(containerName, "-\\.", string.Empty);
containerName = Regex.Replace(containerName, "\\.-", string.Empty);
containerName = Regex.Replace(containerName, "^-", string.Empty);
containerName = Regex.Replace(containerName, "-$", string.Empty);
containerName = Regex.Replace(containerName, "^\\.", string.Empty);
containerName = Regex.Replace(containerName, "\\.$", string.Empty);
containerName = Regex.Replace(containerName, "^(?:(?:^|\\.)(?:2(?:5[0-5]|[0-4]\\d)|1?\\d?\\d)){4}$", String.Empty);
if (containerName.Length < 3)
{
var length = containerName.Length;
for (var i = 0; i < 3 - length; i++)
{
containerName += "0";
}
}
return containerName;
}
}
public virtual string NormalizeBlobName(string blobName)
{
return blobName;
}
}
BlobStoring.Database
该模块为独立模块,管理Blob值存储到数据库中,其内部实现了DatabaseProvider,结合仓储将Blob存储到表中。
扩展
IBlobContainerConfigurationProvider,可以替换或重写默认实现,不再从AbpBlobStoringOptions中直接取BlobContainerConfiguration配置。
IBlobProviderSelector,可以替换或重写默认实现,只需要确保根据ContainerName返回BlobProvider的映射关系,至于内部处理逻辑则完全可以按照业务或技术要求实现。
IBlobContainerFactory,可以替换或重写默认实现,只需按照ContainerName实例化BlobContainer。
IBlobContainer,可以继承该接口,当默认的方法不足够支持业务或技术的需求时,可以结合BlobContainerFactory,增强原有接口功能。
IBlobNamingNormalizer,可以扩展或重写默认实现,以达到期望的命名规范。
2024-08-13,望技术有成后能回来看见自己的脚步。