Distributed execution support of Hive with vineyard #1624
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vegetableysm
force-pushed
the
hive-integration
branch
from
18bfd8b to
3404c55
Compare
vegetableysm
changed the title
vegetableysm
force-pushed
the
hive-integration
branch
from
cec28e6 to
d845938
Compare
|
I try to add location info for splits and Tez is able to get the information I provided, but failed to request a node resource. Here is the log: We can see that yarn task scheduler service get the node hostname of "hadoop-yarn-nm-0" but failed to assign container. When the tez client asks for "/default-rack," it matches the hadoop-yarn-nm-0 node ("not match" in the figure should be " match". It is a mistake). Since the current three nodes are at /default-rack, if the requested resource is "/default-rack", which node the task is assigned to is random. There is no guarantee that the task will be assigned to the node with the corresponding data. Then, I try to create table on HDFS, task allocation also failed. Here is the log: I have no idea why the allocation failed. |
sighingnow
reviewed
Dec 18, 2023
sighingnow
reviewed
Dec 18, 2023
java/modules/basic/src/main/java/io/v6d/modules/basic/arrow/TableBuilder.java
Outdated
Show resolved
Hide resolved
sighingnow
reviewed
Dec 18, 2023
java/hive/src/main/java/io/v6d/hive/ql/io/VineyardInputFormat.java
Outdated
Show resolved
Hide resolved
vegetableysm
force-pushed
the
hive-integration
branch
2 times, most recently
from
172e632 to
b82b67c
Compare
vegetableysm
changed the title
sighingnow
reviewed
Dec 21, 2023
vegetableysm
changed the title
vegetableysm
force-pushed
the
hive-integration
branch
from
11b9784 to
d2ac116
Compare
vegetableysm
changed the title
|
Rebase to main please. |
vegetableysm
force-pushed
the
hive-integration
branch
from
1866920 to
d9a2b8a
Compare
Done |
sighingnow
reviewed
Dec 27, 2023
Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
Supports to creat tables on the vineyard with data distributed across different nodes. Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
Fix a bug of file permission. Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
vegetableysm
force-pushed
the
hive-integration
branch
from
d9a2b8a to
22f2794
Compare
Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>
vegetableysm
force-pushed
the
hive-integration
branch
from
ca95c3e to
d4df239
Compare
sighingnow
approved these changes
Dec 28, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What do these changes do?
Currently, the vineyard jar cannot run directly on hive because of dependency conflicts. You can run it temporarily by reverting to an older version of guava dependent by vineyard.
This problem will be fixed in #1682