[Improvement-17010][DataX] Change datax_channel_count to a custom parameter #17032
Conversation
BruceWong96
requested review from
caishunfeng,
SbloodyS and
Gallardot
as code owners
BruceWong96
changed the title
|
@SbloodyS PTAL. |
...-task-datax/src/main/java/org/apache/dolphinscheduler/plugin/task/datax/DataxParameters.java
Outdated
Show resolved
Hide resolved
Done. Thanks. |
ruanwenjun
force-pushed
the
Improvement-17010
branch
from
93eb4af to
8cf11f7
Compare
Some ci problem can be fixed by #17041 |
@SbloodyS How do i retry analysis of this PR ? |
OK, finished. |
| /** | ||
| * channel count | ||
| */ | ||
| private int channelCount = 1; |
There was a problem hiding this comment.
We need to expose this into UI?
There was a problem hiding this comment.
According to the following documents, the channel can be directly configured in the JSON.
ds-doc-datax
hdfswriter
What do you think about it ?
Thanks.
There was a problem hiding this comment.
If so, this PR seems meanless, it doesn't change the usage.
There was a problem hiding this comment.
If so, this PR seems meanless, it doesn't change the usage.
Sorry, I might have misunderstood this issue.
After reviewing the issue, it’s clear that custom configurations are fully handled through JSON.
Modifying this parameter is only required when using DS’s default configuration.
To address this, we should add a channel configuration option in the UI to let users adjust the channel count setting.
Is my understanding correct?
Thanks.
|
|
|
There was a problem hiding this comment.
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
Comments suppressed due to low confidence (2)
dolphinscheduler-task-plugin/dolphinscheduler-task-datax/src/main/java/org/apache/dolphinscheduler/plugin/task/datax/DataxTask.java:352
- The removal of the hardcoded channel count in buildDataxCoreJson creates a potential inconsistency with buildDataxJobSettingJson, where the channel count is conditionally set. Confirm whether channel count should also be applied in buildDataxCoreJson.
speed.put("channel", DATAX_CHANNEL_COUNT);
dolphinscheduler-task-plugin/dolphinscheduler-task-datax/src/main/java/org/apache/dolphinscheduler/plugin/task/datax/DataxParameters.java:134
- Removal of the custom toString implementation in favor of Lombok’s @DaTa annotation may lead to differences in the string format. Ensure that the test assertions in DataxParametersTest are updated to match Lombok’s default toString output.
public String toString() { ... }
Purpose of the pull request
Change datax_channel_count to a custom parameter.
close #17010
Brief change log
Verify this pull request
This pull request is code cleanup without any test coverage.