1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
|
From 6911c68b96f3a0fc35b40e5c387da51cd84e5461 Mon Sep 17 00:00:00 2001
From: "Neil.wrz" <wangrunze13@huawei.com>
Date: Mon, 13 Feb 2023 03:34:31 -0800
Subject: [PATCH 44/53] doc: add document about support remote ro directory
---
docs/design/README.md | 4 +-
docs/design/README_zh.md | 6 +-
.../design/detailed/Image/remote_ro_design.md | 333 ++++++++++++++++++
3 files changed, 340 insertions(+), 3 deletions(-)
create mode 100644 docs/design/detailed/Image/remote_ro_design.md
diff --git a/docs/design/README.md b/docs/design/README.md
index a9d357b4..cf29c0a1 100644
--- a/docs/design/README.md
+++ b/docs/design/README.md
@@ -34,6 +34,8 @@ This section contains some design documents for users who want to learn more abo
- You can see how the isula search are designed in [image_search_design](./detailed/Image/image_search_design_zh.md)
+- You can see how the remote ro are designed in [remote_ro_design](./detailed/Image/remote_ro_design.md)
+
## Network
- You can see how the cni operator modules are designed in [cni_operator_design](./detailed/Network/cni_operator_design.md).
@@ -54,4 +56,4 @@ This section contains some design documents for users who want to learn more abo
## Volume
-- You can see how the local volume modules are designed in [local_volume_design](./detailed/Volume/local_volume_design.md).
\ No newline at end of file
+- You can see how the local volume modules are designed in [local_volume_design](./detailed/Volume/local_volume_design.md).
diff --git a/docs/design/README_zh.md b/docs/design/README_zh.md
index f51930e4..1f1c94b5 100644
--- a/docs/design/README_zh.md
+++ b/docs/design/README_zh.md
@@ -32,7 +32,9 @@
- 查看 registry 模块的设计文档： [registry_degisn](./detailed/Image/registry_degisn_zh.md) 。
-- 查看 isula search 的设计文档：[image_search_design](./detailed/Image/image_search_design_zh.md)
+- 查看 isula search 的设计文档：[image_search_design](./detailed/Image/image_search_design_zh.md) 。
+
+- 查看 ro目录分离的设计文档： [remote_ro_design](./detailed/Image/remote_ro_design.md) 。
## Network
@@ -54,4 +56,4 @@
## Volume
-- 查看 local volume 模块的设计文档： [local_volume_design](./detailed/Volume/local_volume_design_zh.md).
\ No newline at end of file
+- 查看 local volume 模块的设计文档： [local_volume_design](./detailed/Volume/local_volume_design_zh.md).
diff --git a/docs/design/detailed/Image/remote_ro_design.md b/docs/design/detailed/Image/remote_ro_design.md
new file mode 100644
index 00000000..fee33835
--- /dev/null
+++ b/docs/design/detailed/Image/remote_ro_design.md
@@ -0,0 +1,333 @@
+| Author | 王润泽 |
+| ------ | ---------------------- |
+| Date | 2023-2-13 |
+| Email | wangrunze13@huawei.com |
+
+# 1. 方案目标
+目标有两个：
+1. 把isulad当前的layer store里的RO层分离出来，把RW layer和RO layer分开到不同到目录存储。
+2. isulad在运行时如果在相关目录里恢复了正确的镜像数据(image和layer数据), 可实现不重启isulad, 直接使用新恢复的镜像。如果移除当前没有容器正在使用的镜像数据，可实现不重启isulad，isulad更新当前管理的镜像列表去除该镜像。只考虑完全正确的新增和删除。
+
+
+## 1.1 用法说明
+通过源码编译打开编译选项来开启功能，cmake添加`cmake -DENABLE_REMOTE_LAYER_STORE=ON ..`, 然后`make -j`即可。启动iSulad之前，还需要在配置文件`/etc/isulad/daemon.json`里面添加`"storage-enable-remote-layer": true`来打开开关。
+
+# 2. 总体设计
+
+*Modules Dependencies*
+```
+=> New Added Module:
+
+ +===================================+
+ | Remote Supporter Module |
+ +===================================+
+ | |
+ | +-------------------------+ |
+ | | maintainer submod | |
+ | +-------------------------+ |
+ | | global data initer | |
+ | | symbol link maintainer | |
+ | | global data getter | |
+ | +-------------------------+ |
+ | |
+ | +-------------------------+ |
+ | | Supporter submod | |
+ | +-------------------------+ |
+ | | supporter interface | |
+ | | overlay supporter impl | |
+ | | layer supporter impl | |
+ | | image supporter impl | |
+ | | remote refresh thread | |
+ | +-------------------------+ |
+ +-----------------------------------+
+
+
+=> Modified Modules:
+
+ +===================================+
+ | Storage Module |
+ +===================================+
+ | |
+ | +-------------------------+ |
+ | | Image Store submod | |
+ | | Added Functions | |
+ | +-------------------------+ |
+ | | add image in memory | |
+ | | delete image in memory | |
+ | | get image top layer | |
+ | | valid image manifest | |
+ | +-------------------------+ |
+ | |
+ | +-------------------------+ |
+ | | Layer Store submod | |
+ | | Added Functions | |
+ | +-------------------------+ |
+ | | add layer in memory | |
+ | | delete layer in memory | |
+ | +-------------------------+ |
+ | |
+ | +-------------------------+ |
+ | | Driver Overlay submod | |
+ | | Added Functions | |
+ | +-------------------------+ |
+ | | - | |
+ | +-------------------------+ |
+ +-----------------------------------+
+
+
+=> Modules Dependencies:
+
+ +-------------------------+ +---------------+
+ | Supporter submod | | Image Store |
+ +-------------------------+ +----▶| submod ---------+
+ | supporter interface | | +---------------+ |
+ | overlay supporter impl -----+ |
+ | layer supporter impl -----+ |
++----------------+ | image supporter impl | | +----------------+ |
+| storage module |-------▶| remote refresh thread | +----▶| Layer Store | | init +-----------------------+
++----------------+ +-------------------------+ | submod --------+--------▶| maintainer submod |
+ | +----------------+ | +-----------------------+
+ | | ▲
+ | | |
+ | +----------------+ | |
+ | | Layer Store | | |
+ | | submod --------+ |
+ | +----------------+ |
+ | |
+ +------------------------------------------------------------------------+
+ get global data
+```
+
+
+总体来说有两部分的功能：
+- iSulad原有的image storage适配分离的RO目录结构，*分离的RO目录*可用于远程挂载
+- iSulad实例同步内存数据，镜像数据和layer数据*定期更新*，不通过`isula pull` 和 `isula rmi` 等命令，直接通过分离目录里面的数据来更新镜像数据。
+
+*分离RO目录*
+修改前后storage目录结构对比：
+
+```
+old:
+overlay-layer
+├── b703587
+│ └── layer.json
+└── b64792c
+ └── layer.json
+ └── b64792.tar.gz
+
+new:
+overlay-layer
+├── b64792c -> ../RO/b64792c
+├── b703587
+│ └── layer.json
+└── RO
+ └── b64792c
+ └── layer.json
+ └── b64792.tar.gz
+```
+
+以overlay-layers目录为例，创建新layer时，如果是只读层，就把层数据放到RO目录下，在RO上层目录创建软连接指向真实数据。删除layer时需要额外删除软连接。
+
+
+*定期更新*
+定期更新通过启动一个线程周期扫描`overlay`, `overlay-layers`, `overlay-image`这三个目录，通过比较当前时刻与上一时刻的目录差异，来获取镜像和层的删减情况，进而同步isulad的storage内存数据和维护软链接。
+
+```
++---------------------+ +---------------------+ +---------------------+ +-----------------------+
+| refresh thread loop | | overlay remote impl | | layer remote impl | | image remote impl |
++---------------------+ +---------------------+ +---------------------+ +-----------------------+
+ | | | |
+ | refresh start | | |
+ |-----------------------------▶| | |
+ | | overlay dir scan | |
+ | | | |
+ | | to added layers | |
+ | | memory and symlink add | |
+ | | to deleted layers | |
+ | | memory and symlink del | |
+ | | valid overlay layers | |
+ | |---------------------------------▶| |
+ | | next scan | |
+ | | | |
+ | | | |
+ | | | |
+ | | | overlay-layers dir scan |
+ | | check overlay layer ready | |
+ | |◀---------------------------------| to added layers |
+ | |---------------------------------▶| filter invalid layers |
+ | | result | memory and symlink add |
+ | | | to deleted layers |
+ | | | memory and symlink del |
+ | | | valid overlay layers |
+ | | |---------------------------------▶|
+ | | | next scan |
+ | | | |
+ | | | |
+ | | | |
+ | | | | overlay-image dir scan
+ | | | check layers ready |
+ | | |◀---------------------------------| to added images
+ | | |---------------------------------▶| filter invalid images
+ | | | result | memory add images
+ | | | | to deleted images
+ | | | | memory del images
+ | | | |
+ |◀---------------------------------------------------------------------------------------------------|
+ | refresh end | | |
+ | | | |
++---------------------+ +---------------------+ +---------------------+ +-----------------------+
+| refresh thread loop | | image remote module | | layer remote module | | overlay remote module |
++---------------------+ +---------------------+ +---------------------+ +-----------------------+
+
+```
+
+# 3. 接口描述
+
+```c
+// 初始化remote模块里的layer data
+int remote_layer_init(const char *root_dir);
+
+// 初始化remote模块里的overlay data
+int remote_overlay_init(const char *driver_home);
+
+// 清理remote模块的资源
+void remote_maintain_cleanup();
+
+// 启动 定期更新的thread
+int start_refresh_thread(void);
+
+// 创建新layer目录
+int remote_layer_build_ro_dir(const char *id);
+
+// 创建新overlay目录
+int remote_overlay_build_ro_dir(const char *id);
+
+// 删除layer目录
+int remote_layer_remove_ro_dir(const char *id);
+
+// 删除overlay目录
+int remote_overlay_remove_ro_dir(const char *id);
+```
+
+# 4. 详细设计
+分离RO目录的关键在于适配原来的代码逻辑，原先的代码在操作镜像和层的时候，不管是RO层还是RW层，从创建到删除都是在当前目录下进行的，这就是我们额外创建一个软连接的作用:
+- RO目录的作用是为了支持远程挂载
+- 软连接的作用是模拟原来的目录结构
+
+这样以来，image module的逻辑几乎不需要改动，除了以下几点需要注意：
+- 创建和删除的时候需要处理一个额外的资源：软连接，之前只需要关注目录即可，现在如果创建的是只读层，就需要额外创建软连接，如果删除的是只读层，就需要额外删除软连接
+- 以`overlay-layers`目录为例，isulad启动时会以正则规则扫描当前目录下的子目录是否合法，所以需要屏蔽`RO`目录
+
+定时刷新的逻辑如下：
+以`overlay-image`目录的刷新为例，通过维护两个集合`new` 和 `old`, 这两个集合初始都为空，通过扫描目录里面所有的子目录，把合法的image id 加入`new`集合， 通过计算两个集合的差， 在集合`new`里面存在而在集合`old`里面不存在的id则为新增加的镜像， 在集合`old`里面存在而在集合`new`里面不存在的id则为删除的镜像。处理新增加的镜像还需要额外的一个判断，就是判断镜像的层数据是否已经加载，如果没加载则该镜像本轮不加载。`overlay-layers` 和 `overlay` 目录的处理逻辑类似。
+
+
+*可能的使用场景*
+一个可能的使用场景就是通过远程文件共享(nfs)让多台启动的isulad实例共享某些只读的数据；具体来说，在两个host A和B上都启动了iSulad, 如果A pull或者load了镜像busybox, 那么B上的isulad同样可以使用这个镜像。
+
+```
+operations:
+
++--------------------+ +--------------------+ +--------------------+
+| isula pull busybox | | without pull | | without pull |
+| isula pull nginx | | isula run busybox | | isula run ngxinx |
+| isula pull ... | | isula run ... | | isula run centos |
++--------------------+ +--------------------+ +--------------------+
+ | | |
+ ▼ ▼ ▼
++======================+ +======================+ +======================+
+| isulad on Host A | | isulad on Host B | | isulad on Host C |
++======================+ +======================+ +======================+
+| image store module | | image store module | | image store module |
++----------------------+ +----------------------+ +----------------------+
+| refresh thread off | | refresh thread on | | refresh thread on |
++----------------------+ +----------------------+ +----------------------+
+| local rw | remote ro | | local rw | remote ro | | local rw | remote ro |
++----------------------+ +----------------------+ +----------------------+
+ | | |
+ | enable nfs | mounted on | mounted on
+ ▼ ▼ ▼
+ +=====================================================================+
+ | nfs directory over network |
+ +=====================================================================+
+ | image store |
+ +---------------------------------------------------------------------+
+ | image | image | image | image | image |
+ | busybox | nginx | my-app | ubuntu | centos |
+ | 4MB | 100MB | 1.2GB | 5MB | 6MB |
+ +---------------------------------------------------------------------+
+ | layers store |
+ +---------------------------------------------------------------------+
+ | layer | layer | layer |
+ | 05c361054 | 8a9d75caad | 789d605ac |
+ +---------------------------------------------------------------------+
+```
+
+*同步问题*
+共享资源并发使用发生竞争的条件是：`two process access the same resource concurrently and at least one of the access is a writer`。这里的共享资源有：
+
+```
++============================+ +=====================================+
+| Sharing Resource | | Storage |
++============================+ +=====================================+
+| read-only overlay layers | mounted | /var/lib/isulad/overlay/RO |
++----------------------------+ ======▶ +-------------------------------------+
+| reald-only layers metadata | shared | /var/lib/isulad/overlay-layers/RO |
++----------------------------+ +-------------------------------------+
+| reald-only images | | /var/lib/isulad/overlay-images |
++----------------------------+ +-------------------------------------+
+
+```
+而分布在不同host上的isulad进程通过网络共享这些资源，如果不考虑新增删除的情况，不会出现资源竞争，所有的节点都是reader。
+
+对于主节点pull镜像，其他节点使用镜像的情况，主节点是writer，其他节点是reader。这时候可能出现的问题是主节点pull镜像的流程没有完全结束，但是其他节点开始使用这个不完整的镜像。对于这个问题的解决方案是通过扫描image目录来新增镜像，通过这种方式能确保一定该新增镜像的信息完整。
+
+```
++---------------------+ +--------------------+ +-----------------------+ +-----------------------+
+| registry | | image module | | layer store module | | driver overlay module |
++---------------------+ +--------------------+ +-----------------------+ +-----------------------+
+ | | | |
+ | | | |
+ | registry_pull | | |
+ | fetch_manifest | | |
+ | check reuse and fetch | | |
+ | +----------------+ | | |
+ | | register_layer | | | |
+ | +----------------+ | | |
+ |-----------------------------------------------------------------▶| |
+ | | | layer_store_create |
+ | | |--------------------------------▶|
+ | | | | driver_create_layer
+ | | | | +--------------------+
+ | | | | | setup overlay dir |
+ | | | | +--------------------+
+ | | | | driver_create_layer done
+ | | |◀--------------------------------|
+ | | | +------------+ |
+ | | | | save layer | |
+ | | | +------------+ |
+ | | | layer create done |
+ |◀-----------------------------------------------------------------| |
+ | all layer setup | | |
+ | +----------------+ | | |
+ | | register image | | | |
+ | +----------------+ | | |
+ |-------------------------------▶| | |
+ | | storage_img_create | |
+ | | set img top layer | |
+ | | img create | |
+ | | +------------+ | |
+ | | | save image | | |
+ | | +------------+ | |
+ | | create image done | |
+ |◀-------------------------------| | |
+ | pull img done | | |
+ | | | |
+ | | | |
++---------------------+ +--------------------+ +-----------------------+ +-----------------------+
+| registry | | image module | | layer store module | | driver overlay module |
++---------------------+ +--------------------+ +-----------------------+ +-----------------------+
+
+```
+
+至于主节点删除镜像的情况，主节点是writer，其他节点是reader，可能出现的情况是其他节点还有容器在使用镜像的时候，镜像被删除，但是根据需求场景暂不处理这种情况。其他的处理与新增镜像相同，依然以image dir作为入口，扫描发现删除的镜像。删除时需要关注layer和overlay目录下的软链接。
--
2.25.1
|
