首页 注册 登录
V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
V2EX 提问指南
广告
Eremita
V2EX  ›  问与答

使用 eclipse 开发 hadoop1.2.1 时， io.sort.mp 怎么更改？

  •   
  •   Eremita · 2017-04-07 12:53:01 +08:00 · 1409 次点击
    这是一个创建于 2788 天前的主题，其中的信息可能已经有所发展或是发生改变。

    问题如题，我想增加每个任务的内存限制，但是不知道怎么正确配置，希望有人可以帮助我。

    hadoop/conf

    core_site.xml 中，我设置了

    	<property>
		<name>io.sort.mb</name>
		<value>3000</value>
	</property>

	<property>
		<name>io.sort.factor</name>
		<value>300</value>
	</property>

    eclipse

    在 Edit Hadoop location 的 Advanced parameters 中，我也配置 io.sort.factor 和 io.sort.mb ，但是运行时还是默认的 100

    运行结果

    
17/04/07 12:43:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/07 12:43:22 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
17/04/07 12:43:22 INFO input.FileInputFormat: Total input paths to process : 1
17/04/07 12:43:23 INFO mapred.JobClient: Running job: job_local805018878_0001
17/04/07 12:43:23 INFO mapred.LocalJobRunner: Waiting for map tasks
17/04/07 12:43:23 INFO mapred.LocalJobRunner: Starting task: attempt_local805018878_0001_m_000000_0
17/04/07 12:43:23 INFO util.ProcessTree: setsid exited with exit code 0
17/04/07 12:43:23 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@58948e69
17/04/07 12:43:23 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/ubuntu/input/FiveWords2EightWords.data:0+67108864
17/04/07 12:43:23 INFO mapred.MapTask: io.sort.mb = 100
17/04/07 12:43:23 INFO mapred.MapTask: data buffer = 79691776/99614720
17/04/07 12:43:23 INFO mapred.MapTask: record buffer = 262144/327680
17/04/07 12:43:24 INFO mapred.JobClient:  map 0% reduce 0%
17/04/07 12:43:25 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:25 INFO mapred.MapTask: bufstart = 0; bufend = 4194304; bufvoid = 99614720
17/04/07 12:43:25 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
17/04/07 12:43:25 INFO mapred.MapTask: Finished spill 0
17/04/07 12:43:27 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:27 INFO mapred.MapTask: bufstart = 4194304; bufend = 8388592; bufvoid = 99614720
17/04/07 12:43:27 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
17/04/07 12:43:27 INFO mapred.MapTask: Starting flush of map output
17/04/07 12:43:28 INFO mapred.MapTask: Finished spill 1
17/04/07 12:43:28 INFO mapred.MapTask: Finished spill 2
17/04/07 12:43:28 INFO mapred.Merger: Merging 3 sorted segments
17/04/07 12:43:28 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 9437190 bytes
17/04/07 12:43:28 INFO mapred.Task: Task:attempt_local805018878_0001_m_000000_0 is done. And is in the process of commiting
17/04/07 12:43:28 INFO mapred.LocalJobRunner: 
17/04/07 12:43:28 INFO mapred.Task: Task 'attempt_local805018878_0001_m_000000_0' done.
17/04/07 12:43:28 INFO mapred.LocalJobRunner: Finishing task: attempt_local805018878_0001_m_000000_0
17/04/07 12:43:28 INFO mapred.LocalJobRunner: Starting task: attempt_local805018878_0001_m_000001_0
17/04/07 12:43:28 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@742efee7
17/04/07 12:43:28 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/ubuntu/input/FiveWords2EightWords.data:67108864+67108864
17/04/07 12:43:28 INFO mapred.MapTask: io.sort.mb = 100
17/04/07 12:43:28 INFO mapred.MapTask: data buffer = 79691776/99614720
17/04/07 12:43:28 INFO mapred.MapTask: record buffer = 262144/327680
17/04/07 12:43:29 INFO mapred.JobClient:  map 20% reduce 0%
17/04/07 12:43:29 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:29 INFO mapred.MapTask: bufstart = 0; bufend = 4194304; bufvoid = 99614720
17/04/07 12:43:29 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
17/04/07 12:43:29 INFO mapred.MapTask: Finished spill 0
17/04/07 12:43:31 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:31 INFO mapred.MapTask: bufstart = 4194304; bufend = 8388592; bufvoid = 99614720
17/04/07 12:43:31 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
17/04/07 12:43:31 INFO mapred.MapTask: Starting flush of map output
17/04/07 12:43:31 INFO mapred.MapTask: Finished spill 1
17/04/07 12:43:31 INFO mapred.MapTask: Finished spill 2
17/04/07 12:43:31 INFO mapred.Merger: Merging 3 sorted segments
17/04/07 12:43:31 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 9437190 bytes
17/04/07 12:43:31 INFO mapred.Task: Task:attempt_local805018878_0001_m_000001_0 is done. And is in the process of commiting
17/04/07 12:43:31 INFO mapred.LocalJobRunner: 
17/04/07 12:43:31 INFO mapred.Task: Task 'attempt_local805018878_0001_m_000001_0' done.
17/04/07 12:43:31 INFO mapred.LocalJobRunner: Finishing task: attempt_local805018878_0001_m_000001_0
17/04/07 12:43:31 INFO mapred.LocalJobRunner: Starting task: attempt_local805018878_0001_m_000002_0
17/04/07 12:43:31 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@42c39ed
17/04/07 12:43:31 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/ubuntu/input/FiveWords2EightWords.data:134217728+67108864
17/04/07 12:43:31 INFO mapred.MapTask: io.sort.mb = 100
17/04/07 12:43:31 INFO mapred.MapTask: data buffer = 79691776/99614720
17/04/07 12:43:31 INFO mapred.MapTask: record buffer = 262144/327680
17/04/07 12:43:32 INFO mapred.JobClient:  map 40% reduce 0%
17/04/07 12:43:33 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:33 INFO mapred.MapTask: bufstart = 0; bufend = 4194304; bufvoid = 99614720
17/04/07 12:43:33 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
17/04/07 12:43:33 INFO mapred.MapTask: Finished spill 0
17/04/07 12:43:34 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:34 INFO mapred.MapTask: bufstart = 4194304; bufend = 8388592; bufvoid = 99614720
17/04/07 12:43:34 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
17/04/07 12:43:34 INFO mapred.MapTask: Starting flush of map output
17/04/07 12:43:34 INFO mapred.MapTask: Finished spill 1
17/04/07 12:43:34 INFO mapred.MapTask: Finished spill 2
17/04/07 12:43:34 INFO mapred.Merger: Merging 3 sorted segments
17/04/07 12:43:34 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 9437190 bytes
17/04/07 12:43:34 INFO mapred.Task: Task:attempt_local805018878_0001_m_000002_0 is done. And is in the process of commiting
17/04/07 12:43:34 INFO mapred.LocalJobRunner: 
17/04/07 12:43:34 INFO mapred.Task: Task 'attempt_local805018878_0001_m_000002_0' done.
17/04/07 12:43:34 INFO mapred.LocalJobRunner: Finishing task: attempt_local805018878_0001_m_000002_0
17/04/07 12:43:34 INFO mapred.LocalJobRunner: Starting task: attempt_local805018878_0001_m_000003_0
17/04/07 12:43:34 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@16b56343
17/04/07 12:43:34 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/ubuntu/input/FiveWords2EightWords.data:201326592+67108864
17/04/07 12:43:34 INFO mapred.MapTask: io.sort.mb = 100
17/04/07 12:43:34 INFO mapred.MapTask: data buffer = 79691776/99614720
17/04/07 12:43:34 INFO mapred.MapTask: record buffer = 262144/327680
17/04/07 12:43:35 INFO mapred.JobClient:  map 60% reduce 0%
17/04/07 12:43:36 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:36 INFO mapred.MapTask: bufstart = 0; bufend = 4194304; bufvoid = 99614720
17/04/07 12:43:36 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
17/04/07 12:43:36 INFO mapred.MapTask: Finished spill 0
17/04/07 12:43:37 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:37 INFO mapred.MapTask: bufstart = 4194304; bufend = 8388592; bufvoid = 99614720
17/04/07 12:43:37 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
17/04/07 12:43:37 INFO mapred.MapTask: Starting flush of map output
17/04/07 12:43:37 INFO mapred.MapTask: Finished spill 1
17/04/07 12:43:37 INFO mapred.MapTask: Finished spill 2
17/04/07 12:43:37 INFO mapred.Merger: Merging 3 sorted segments
17/04/07 12:43:37 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 9437190 bytes
17/04/07 12:43:37 INFO mapred.Task: Task:attempt_local805018878_0001_m_000003_0 is done. And is in the process of commiting
17/04/07 12:43:37 INFO mapred.LocalJobRunner: 
17/04/07 12:43:37 INFO mapred.Task: Task 'attempt_local805018878_0001_m_000003_0' done.
17/04/07 12:43:37 INFO mapred.LocalJobRunner: Finishing task: attempt_local805018878_0001_m_000003_0
17/04/07 12:43:37 INFO mapred.LocalJobRunner: Starting task: attempt_local805018878_0001_m_000004_0
17/04/07 12:43:37 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4b9c21a4
17/04/07 12:43:37 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/ubuntu/input/FiveWords2EightWords.data:268435456+67108864
17/04/07 12:43:37 INFO mapred.MapTask: io.sort.mb = 100
17/04/07 12:43:37 INFO mapred.MapTask: data buffer = 79691776/99614720
17/04/07 12:43:37 INFO mapred.MapTask: record buffer = 262144/327680
17/04/07 12:43:38 INFO mapred.JobClient:  map 80% reduce 0%
17/04/07 12:43:38 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:38 INFO mapred.MapTask: bufstart = 0; bufend = 4194304; bufvoid = 99614720
17/04/07 12:43:38 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
17/04/07 12:43:38 INFO mapred.MapTask: Finished spill 0
17/04/07 12:43:40 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:40 INFO mapred.MapTask: bufstart = 4194304; bufend = 8388592; bufvoid = 99614720
17/04/07 12:43:40 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
17/04/07 12:43:40 INFO mapred.MapTask: Starting flush of map output
17/04/07 12:43:40 INFO mapred.MapTask: Finished spill 1
17/04/07 12:43:40 INFO mapred.MapTask: Finished spill 2
17/04/07 12:43:40 INFO mapred.Merger: Merging 3 sorted segments
17/04/07 12:43:40 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 9437190 bytes
17/04/07 12:43:40 INFO mapred.Task: Task:attempt_local805018878_0001_m_000004_0 is done. And is in the process of commiting
17/04/07 12:43:40 INFO mapred.LocalJobRunner: 
17/04/07 12:43:40 INFO mapred.Task: Task 'attempt_local805018878_0001_m_000004_0' done.
17/04/07 12:43:40 INFO mapred.LocalJobRunner: Finishing task: attempt_local805018878_0001_m_000004_0
17/04/07 12:43:40 INFO mapred.LocalJobRunner: Map task executor complete.
17/04/07 12:43:40 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@24feb67c
17/04/07 12:43:40 INFO mapred.LocalJobRunner: 
17/04/07 12:43:40 INFO mapred.Merger: Merging 5 sorted segments
17/04/07 12:43:40 INFO mapred.Merger: Down to the last merge-pass, with 5 segments left of total size: 47185930 bytes
17/04/07 12:43:40 INFO mapred.LocalJobRunner: 
17/04/07 12:43:41 INFO mapred.JobClient:  map 100% reduce 0%
17/04/07 12:43:43 INFO mapred.Task: Task:attempt_local805018878_0001_r_000000_0 is done. And is in the process of commiting
17/04/07 12:43:43 INFO mapred.LocalJobRunner: 
17/04/07 12:43:43 INFO mapred.Task: Task attempt_local805018878_0001_r_000000_0 is allowed to commit now
17/04/07 12:43:43 INFO output.FileOutputCommitter: Saved output of task 'attempt_local805018878_0001_r_000000_0' to hdfs://localhost:9000/user/ubuntu/output
17/04/07 12:43:43 INFO mapred.LocalJobRunner: reduce > reduce
17/04/07 12:43:43 INFO mapred.Task: Task 'attempt_local805018878_0001_r_000000_0' done.
17/04/07 12:43:44 INFO mapred.JobClient:  map 100% reduce 100%
17/04/07 12:43:44 INFO mapred.JobClient: Job complete: job_local805018878_0001
17/04/07 12:43:44 INFO mapred.JobClient: Counters: 22
17/04/07 12:43:44 INFO mapred.JobClient:   Map-Reduce Framework
17/04/07 12:43:44 INFO mapred.JobClient:     Spilled Records=7864320
17/04/07 12:43:44 INFO mapred.JobClient:     Map output materialized bytes=47185950
17/04/07 12:43:44 INFO mapred.JobClient:     Reduce input records=2621440
17/04/07 12:43:44 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=0
17/04/07 12:43:44 INFO mapred.JobClient:     Map input records=2621440
17/04/07 12:43:44 INFO mapred.JobClient:     SPLIT_RAW_BYTES=650
17/04/07 12:43:44 INFO mapred.JobClient:     Map output bytes=41943040
17/04/07 12:43:44 INFO mapred.JobClient:     Reduce shuffle bytes=0
17/04/07 12:43:44 INFO mapred.JobClient:     Physical memory (bytes) snapshot=0
17/04/07 12:43:44 INFO mapred.JobClient:     Reduce input groups=2621440
17/04/07 12:43:44 INFO mapred.JobClient:     Combine output records=0
17/04/07 12:43:44 INFO mapred.JobClient:     Reduce output records=2621440
17/04/07 12:43:44 INFO mapred.JobClient:     Map output records=2621440
17/04/07 12:43:44 INFO mapred.JobClient:     Combine input records=0
17/04/07 12:43:44 INFO mapred.JobClient:     CPU time spent (ms)=0
17/04/07 12:43:44 INFO mapred.JobClient:     Total committed heap usage (bytes)=24697110528
17/04/07 12:43:44 INFO mapred.JobClient:   File Input Format Counters 
17/04/07 12:43:44 INFO mapred.JobClient:     Bytes Read=335544320
17/04/07 12:43:44 INFO mapred.JobClient:   FileSystemCounters
17/04/07 12:43:44 INFO mapred.JobClient:     HDFS_BYTES_READ=1342177280
17/04/07 12:43:44 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=377897656
17/04/07 12:43:44 INFO mapred.JobClient:     FILE_BYTES_READ=235943170
17/04/07 12:43:44 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=33910562
17/04/07 12:43:44 INFO mapred.JobClient:   File Output Format Counters 
17/04/07 12:43:44 INFO mapred.JobClient:     Bytes Written=33910562
    目前尚无回复
    mapredinfomaptasktask
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   2769 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 23ms · UTC 08:19 · PVG 16:19 · LAX 00:19 · JFK 03:19
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.