V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
TomeWong
V2EX  ›  程序员

electron 使用 web worker 上传批量的文件(7W+)报 Maximum call stack size exceeded 错误

  •  
  •   TomeWong · 2020-04-29 10:56:37 +08:00 · 1863 次点击
    这是一个创建于 1660 天前的主题,其中的信息可能已经有所发展或是发生改变。

    web worker 从主进程获取到上传文件的路径,在 onmessage 处理上传,通过 fs.promises.stat 获取文件的信息,通过 postMessage 回传至 worker 主进程中,在 worker 主进程中,将这些数据更新至 vuex 中,出现了下面的问题,小数量级的文件,不会出现这个问题

      (node:33176) UnhandledPromiseRejectionWarning: RangeError: Maximum call stack size exceeded
          at Array.mutator (D:\pc-win\SKYD-win\node_modules\vue\dist\vue.runtime.common.dev.js:870:27)
          at Store.addMultiTaskUL (webpack:///./src/renderer/store/modules/task.js?:164:72)
          at wrappedMutationHandler (D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:727:13)
          at commitIterator (D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:393:7)
          at Array.forEach (<anonymous>)
          at D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:392:11
          at Store._withCommit (D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:523:3)
          at Store.commit (D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:391:8)
          at Store.boundCommit [as commit] (D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:336:19)
          at local.commit (D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:681:13)
          at eval (webpack:///./src/renderer/store/modules/task.js?:476:7)
          at new Promise (<anonymous>)
          at new F (webpack:///./node_modules/core-js/library/modules/_export.js?:36:28)
          at Store.addMultiTaskUL (webpack:///./src/renderer/store/modules/task.js?:440:12)
          at Array.wrappedActionHandler (D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:734:23)
          at Store.dispatch (D:\pc-win\SKYD-win\node_modules\vuex\dist\vuex.common.js:439:15)
      (node:33176) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 10)
    
    10 条回复    2020-04-29 15:11:08 +08:00
    TomeWong
        1
    TomeWong  
    OP
       2020-04-29 11:10:42 +08:00
    ```
    self.onmessage = async event => {
    console.log(event)
    const { diskType, files, dirpath, sid, usn, currentFileId, currentGroupId, currentCreatedByUsn, baseUrl, uploadLimit, uploadSpeed } = event.data
    const queueData = [],
    taskDataArr = []
    let completeStatus
    for (let elem of files) {
    const filePath = elem,
    // stats = await fs.promises.stat(filePath).catch(error => console.log(error)),
    // size = stats.size,
    chunkSize = 5 * 1024 * 1024, //每片分块的大小 5M
    // pieces = Math.ceil(size / chunkSize),
    uploadType = pieces == 1 ? 1 : 3,
    num = filePath.lastIndexOf('\\') + 1,
    name = filePath.substring(num),
    uuid = uuidv4(),
    basename = dirpath ? path.basename(dirpath) : '',
    realPath = dirpath ? filePath.substring(filePath.indexOf(basename), filePath.lastIndexOf("\\")).replace(/\\/g, "/") : ''
    const payload = {
    appFileId: "",
    diskType: diskType,
    uploadType: uploadType,
    creatorUsn: currentCreatedByUsn || usn,
    parentid: currentFileId == -2 ? -1 : currentFileId || -1,
    groupId: diskType == 1 ? '' : currentGroupId,
    fileSize: 0,
    fileName: name,
    fileMd5: '',
    fileRealPath: realPath,
    comeFrom: 30,
    model: 0,
    discussContent: ""
    }
    const queueItemData = { sid, uuid, payload, name, size, currentFileId, uploadType, pieces, chunkSize, filePath, currentCreatedByUsn, baseUrl, uploadLimit, uploadSpeed }
    // taskDataArr.push({ filePath, name, uuid, size, pieces, uploadType, diskType, parentid: currentFileId == -2 ? -1 : currentFileId || -1, fileRealPath: realPath })
    taskDataArr.push({ filePath, name, uuid, size, pieces, uploadType, diskType, parentid: currentFileId == -2 ? -1 : currentFileId || -1, fileRealPath: realPath })
    queueData.push(queueItemData)

    }
    completeStatus = {
    status: 'updateStore'
    }
    self.postMessage({taskDataArr, completeStatus})
    }
    ```
    TomeWong
        2
    TomeWong  
    OP
       2020-04-29 11:23:57 +08:00
    是否转换字符串来传输,而不是数组对象
    wednesdayco
        3
    wednesdayco  
       2020-04-29 13:59:28 +08:00
    分……片……
    TomeWong
        4
    TomeWong  
    OP
       2020-04-29 14:27:13 +08:00
    @wednesdayco 现在都是小文件,可以正常上传,但将 web worker 中处理的信息回传至 worker 主进程,更新至 vuex 有问题,vuex 是不是存在存储大小限制的问题
    Vegetable
        5
    Vegetable  
       2020-04-29 14:30:01 +08:00
    显然这是递归层数过多
    你是一个一个上传, 一个文件成功之后, 回调里传下一个吗?
    Vegetable
        6
    Vegetable  
       2020-04-29 14:32:31 +08:00
    let deep = 1

    function c() {
    deep++
    c()
    }

    try {
    c()
    } catch (error) {
    console.log(error)
    //RangeError: Maximum call stack size exceeded
    console.log(deep)
    //15712
    }
    TomeWong
        7
    TomeWong  
    OP
       2020-04-29 14:39:58 +08:00
    @Vegetable 是一个一个上传的,并发量为 3,上传是正常的,在上传前会先根据每条文件的路径获取一些信息,然后再将这些信息通过 postMessage 回传至 worker 主进程,在 worker 主进程将这些处理的信息更新至 vuex 中,这个更新过程出现了问题
    VDimos
        8
    VDimos  
       2020-04-29 14:40:39 +08:00 via Android
    你这递归层级太多了吧,做下尾递归优化,不要保留递归栈。
    MrYELiex
        9
    MrYELiex  
       2020-04-29 14:41:00 +08:00
    错误信息很明显 调用栈过多
    TomeWong
        10
    TomeWong  
    OP
       2020-04-29 15:11:08 +08:00
    @VDimos 在 vuex 中使用 map 来处理,可能会有问题
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   1282 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 24ms · UTC 18:06 · PVG 02:06 · LAX 10:06 · JFK 13:06
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.