Assuming, like you said, that you have segmented out the upper left black part all by itself. Let's call that "binaryImage". Then you can just run bwboundaries to get the coordinates.
boundaries = bwboundaries(binaryImage);
x = bwboundaries{1}(:,1); % Think this is the right syntax but not sure.
y = bwboundaries{1}(:,2); % Think this is the right syntax but not sure.
Then find any x values that are 1 and remove those coordinates. Then find any y values that are 1 and remove those points too. Now you have just the ragged points. Pass them into polyfit
coefficients = polyfit(x, y, 1);
Now find out where the line intersects the y=1 and x=1 edges. Then construct a triangle from the 3 coordinates and use poly2mask to create a binary image.
mask = poly2mask(triX, tryY, rows, columns);
Then blacken above the line my masking
newBinaryImage = binaryImage;
newBinaryImage(mask) = false;
However about half of the ragged black part will be below the fitted line and you'll need to fill those in. This can be done by finding them by ANDing your original image with the inverse of the triangle mask.
raggedSpots = binaryImage & ~mask;
Now fill them in in the newBinaryImage image.
newBinaryImage(raggedSpots) = true;
This was totally just off the top of my head and untested. There's still a little for you to do, I'm sure. Come back with your code and original image if you need me to debug it for you. Good luck.
